Facilitating human intervention in an autonomous device

ABSTRACT

Methods, apparatuses, systems, and computer program products for facilitating human intervention in an autonomous device are disclosed. In a particular embodiment, a method of facilitating human intervention in an autonomous device includes a service controller selecting from a first plurality of human interventionists, by a service controller, a first set of human interventionists to respond to a request associated with an autonomous device; transmitting, by the service controller, the request to a first set of interventionist devices, each interventionist device of the first set of interventionist devices associated with a particular human interventionist in the first set of human interventionists; and receiving from the first set of interventionist devices, by the service controller, a first set of interventionist responses to the request.

BACKGROUND

An autonomous device is a device that performs some behaviors or tasksin an autonomous manner, without requiring human control or direction.In some instances, while performing an operation in an autonomousmanner, the autonomous device may determine that the internalinstructions/routines/processes of the autonomous device are incapableof successfully a) executing a given task, b) identifying a next task toperform, or c) judging whether a task was successfully completed,related to both a) and b), and c) either entirely, or with a requireddegree of certainty or confidence.

SUMMARY

Methods, apparatuses, systems, and computer program products forfacilitating human intervention in an autonomous device are disclosed.In a particular embodiment, a method of facilitating human interventionin an autonomous device includes a service controller selecting from afirst plurality of human interventionists a first set of humaninterventionists to respond to a request associated with an autonomousdevice; transmitting the request to a first set of interventionistdevices, each interventionist device of the first set of interventionistdevices associated with a particular human interventionist in the firstset of human interventionists; and receiving from the first set ofinterventionist devices, a first set of interventionist responses to therequest.

In a particular embodiment, an apparatus for facilitating humanintervention in an autonomous device includes a computer processorcoupled to memory. In this embodiment, the memory includes computerprogram instructions that when executed by the computer processor causethe apparatus to carry out the operations of for each humaninterventionist of a plurality of human interventionists: transmitting,by a service controller, a test request to a particular interventionistdevice associated with a particular human interventionist; receivingfrom the particular interventionist device, by the service controller, atest response to the transmitted test request; and based on the testresponse, generating, by the service controller, a performance metric ofthe human interventionist; and based on the performance metrics of thehuman interventionists of the plurality of human interventionists,determining, by the service controller, a rating for each humaninterventionist of the plurality of human interventionists.

In a particular embodiment, a method for facilitating human interventionin an autonomous device includes the autonomous device placing an objectin a stack of one or more objects; in response to placing the object inthe stack, detecting instability in the stack; determining thatcorrecting the instability requires a level of capability beyond athreshold associated with the autonomous device; and in response todetermining that correcting the instability requires the level ofcapability beyond the threshold associated with the autonomous device,generating a request for human intervention.

In a particular embodiment, a method for facilitating human interventionin an autonomous device includes the autonomous device grasping anobject with one or more components of the autonomous device; based onsensor input, determining a level of confidence that the object issecurely grasped with the one or more components of the autonomousdevice; determining that the level does not satisfy a confidencerequirement; and in response to determining that the level does notsatisfy the confidence requirement, generating a request for humanintervention.

In a particular embodiment, a method for facilitating human interventionin an autonomous device includes a service controller receiving arequest from an autonomous device; generating a service response to therequest. In this embodiment, the service response generated based onleast one of: a first set of interventionist responses received from afirst set of interventionist devices associated with a first set ofhuman interventionists; and a first set of autonomous responsesgenerated by an autonomous module. The method also includes training, bythe service controller, the autonomous module using at least one of thefirst set of interventionist responses and the first set of autonomousresponses.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 sets forth a diagram of a system configured for facilitatinghuman intervention in an autonomous device according to embodiments ofthe present disclosure;

FIG. 2 sets forth a flowchart of an example method for facilitatinghuman intervention in an autonomous device according to embodiments ofthe present disclosure;

FIG. 3 sets forth a flowchart of another example method for facilitatinghuman intervention in an autonomous device according to embodiments ofthe present disclosure;

FIG. 4 sets forth a flowchart of another example method for facilitatinghuman intervention in an autonomous device according to embodiments ofthe present disclosure;

FIG. 5A sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 5B sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 6 sets forth a flowchart of another example method for facilitatinghuman intervention in an autonomous device according to embodiments ofthe present disclosure;

FIG. 7A sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 7B sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 8 sets forth a flowchart of another example method for facilitatinghuman intervention in an autonomous device according to embodiments ofthe present disclosure;

FIG. 9 sets forth a flowchart of another example method for facilitatinghuman intervention in an autonomous device according to embodiments ofthe present disclosure;

FIG. 10 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 11 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 12 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 13 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 14 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 15 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 16 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 17 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 18 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 19A sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 19B sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 20 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 21 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 22 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 23 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure;

FIG. 24 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure; and

FIG. 25 sets forth a flowchart of another example method forfacilitating human intervention in an autonomous device according toembodiments of the present disclosure.

DETAILED DESCRIPTION

Particular aspects of the present disclosure are described below withreference to the drawings. In the description, common features aredesignated by common reference numbers throughout the drawings. As usedherein, various terminology is used for the purpose of describingparticular implementations only and is not intended to be limiting. Forexample, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It may be further understood that the terms “comprise,”“comprises,” and “comprising” may be used interchangeably with“include,” “includes,” or “including.” Additionally, it will beunderstood that the term “wherein” may be used interchangeably with“where.” As used herein, “exemplary” may indicate an example, animplementation, and/or an aspect, and should not be construed aslimiting or as indicating a preference or a preferred implementation. Asused herein, an ordinal term (e.g., “first,” “second,” “third,” etc.)used to modify an element, such as a structure, a component, anoperation, etc., does not by itself indicate any priority or order ofthe element with respect to another element, but rather merelydistinguishes the element from another element having a same name (butfor use of the ordinal term). As used herein, the term “set” refers to agrouping of one or more elements, and the term “plurality” refers tomultiple elements.

In the present disclosure, terms such as “determining,” “calculating,”“estimating,” “shifting,” “adjusting,” etc. may be used to describe howone or more operations are performed. It should be noted that such termsare not to be construed as limiting and other techniques may be utilizedto perform similar operations. Additionally, as referred to herein,“generating,” “calculating,” “estimating,” “using,” “selecting,”“accessing,” and “determining” may be used interchangeably. For example,“generating,” “calculating,” “estimating,” or “determining” a parameter(or a signal) may refer to actively generating, estimating, calculating,or determining the parameter (or the signal) or may refer to using,selecting, or accessing the parameter (or signal) that is alreadygenerated, such as by another component or device.

As used herein, “coupled” may include “communicatively coupled,”“electrically coupled,” or “physically coupled,” and may also (oralternatively) include any combinations thereof. Two devices (orcomponents) may be coupled (e.g., communicatively coupled, electricallycoupled, or physically coupled) directly or indirectly via one or moreother devices, components, wires, buses, networks (e.g., a wirednetwork, a wireless network, or a combination thereof), etc. Two devices(or components) that are electrically coupled may be included in thesame device or in different devices and may be connected viaelectronics, one or more connectors, or inductive coupling, asillustrative, non-limiting examples. In some implementations, twodevices (or components) that are communicatively coupled, such as inelectrical communication, may send and receive electrical signals(digital signals or analog signals) directly or indirectly, such as viaone or more wires, buses, networks, etc. As used herein, “directlycoupled” may include two devices that are coupled (e.g., communicativelycoupled, electrically coupled, or physically coupled) withoutintervening components.

Exemplary methods, apparatuses, and computer program products forfacilitating human intervention in an autonomous device in accordancewith the present invention are described with reference to theaccompanying drawings, beginning with FIG. 1. FIG. 1 sets forth adiagram of a system (100) configured for facilitating human interventionin an autonomous device according to embodiments of the presentdisclosure. The system (100) of FIG. 1 includes an interventionistservice server (101), an autonomous device (111), and an interventionistdevice (121).

The autonomous device (111) may be any type of device or machineconfigured for autonomous or semi-autonomous operations and facilitatinghuman intervention according to embodiments of the present disclosure.Examples of autonomous devices include but are not limited toautomobiles, air vehicles, drones, robots, and other as will occur tothose of skill in the art.

In the example of FIG. 1, the autonomous device (111) includes aprocessor (112) coupled to a memory (113), communication devices (119),operation devices (114), input devices (115), and output devices (116).Operation devices may include circuitry, hardware, software, or anycombination thereof for performing the various operations of theautonomous device. The types of operation devices of an autonomousdevice may be dependent upon the type of the autonomous device. Forexample, the operation devices of a UAV may include motors, rotors, andactuators for controlling the flight operations of the UAV. As anotherexample, the operation devices of a manufacturing robot may includewheel controls, swivel controls, cable winding controls, and shovelcontrols.

The input devices (115) may include circuitry, hardware, software, orany combination thereof for capturing, receiving, or otherwise inputtinginformation into the autonomous device. Examples of input devicesinclude but are not limited to cameras, microphones, keyboards, keypads,touchpads, light sensors, radar, positioning circuitry, pressuresensors, fluid sensors, temperature sensors, accelerometers, wheel speedsensors, gyroscopes, altimeters, and many others as will occur to thoseof skill in the art.

The input devices (115) are coupled to the processor (112), such thatdata from the input devices may be available for use by instructionsexecuted by the processor. For example, the input devices may include acamera that is part of a computer vision system. In this example, thecamera may capture images or video to be used by the processor duringperformance of one or more operations, such as a landing operation, atakeoff operation, or object/collision avoidance, as non-limitingexamples.

The output devices (116) may include circuitry, hardware, software, orany combination thereof for presenting, displaying, relaying, orotherwise outputting information from the autonomous device. Examples ofoutput devices include but are not limited to display screens, speakers,tactile feedback devices, and many others as will occur to those ofskill in the art. The output devices (116) are coupled to the processor(112), such that data from instructions executed by the processor may berelated to the output devices.

The communication devices (119) of the autonomous device (111) mayinclude circuitry, hardware, software, or any combination thereof forcommunicating with another device including, for example, transmittingand receiving data. Examples of communication devices may include butare not limited to wireless and wired devices, such as radio frequency(RF) transceivers, Bluetooth transceivers, ethernet controllers,cellular controllers, and wi-fi controllers. In the example of FIG. 1,the connection between the autonomous device (111) and the network (130)is illustrated as a wireless connection. However, readers of skill inthe art will realize that the autonomous device (111) may be coupled tothe network (130) or another device via a wired connection.

The memory (113) of the autonomous device (111) includes operationinstructions (118) comprising computer program instructions that whenexecuted by the processor (112) cause the processor to control theoperation devices (114), the input devices (115), and the output devices(116). For example, if the autonomous device (111) is a UAV, theoperation instructions may control flight control operations, such asadjusting the speed of motors and actuators. Continuing with thisexample, the operation instructions may also control the capturing ofvideo from a camera input device on the UAV and relaying of the capturedvideo via the communication device.

The memory (113) of the autonomous device (111) also includes anautonomous module (120) that includes computer program instructions thatwhen executed by the processor (112) cause the processor (112) toperform instructions/processes/operations autonomously. For example, arobot may include instructions for maneuvering without humanintervention.

In the example of FIG. 1, the memory (113) of the autonomous device(111) also includes an intervention controller (117) that includescomputer program instructions for facilitating human intervention in anautonomous device. Specifically, the intervention controller (117)includes computer program instructions that when executed by theprocessor (112) cause the processor (112) to: generate a request inresponse to detecting an interrupt to an autonomous operation; transmitthe request to the interventionist service server; receive a response tothe request from the interventionist service server; and perform anoperation in accordance with the guidance indicated in the receivedresponse.

The interventionist device (121) may be any type of device or machineconfigured for interfacing with a human and facilitating humanintervention in an autonomous device according to embodiments of thepresent disclosure. Examples of interventionist devices include but arenot limited to mobile devices, smartphones, tablets, laptops, computers,touch controllers, keypads, haptic devices, virtual reality (VR)headsets and systems, and others as will occur to those of skill in theart.

In the example of FIG. 1, the interventionist device (121) includes aprocessor (122) coupled to a memory (123), communication devices (129),operation devices (124), input devices (125), and output devices (126).Operation devices may include circuitry, hardware, software, or anycombination thereof for performing the various operations of theinterventionist device. The types of operation devices of aninterventionist device may be dependent upon the type of theinterventionist device.

The input devices (125) may include circuitry, hardware, software, orany combination thereof for capturing, receiving, or otherwise inputtinginformation into the interventionist device. Examples of input devicesinclude but are not limited to cameras, microphones, keyboards, keypads,touchpads, light sensors, radar, positioning circuitry, pressuresensors, fluid sensors, temperature sensors, accelerometers, wheel speedsensors, gyroscopes, altimeters, and many others as will occur to thoseof skill in the art.

The input devices (125) are coupled to the processor (122), such thatdata from the input devices may be available for use by instructionsexecuted by the processor. For example, the input devices may include acamera that is part of a computer vision system. In this example, thecamera may capture images or video to be used by the processor duringperformance of one or more operations, such as detecting movement of ahuman.

The output devices (126) may include circuitry, hardware, software, orany combination thereof for presenting, displaying, relaying, orotherwise outputting information from the interventionist device.Examples of output devices include but are not limited to displayscreens, speakers, tactile feedback devices, and many others as willoccur to those of skill in the art. The output devices (126) are coupledto the processor (122), such that data from instructions executed by theprocessor may be related to the output devices.

The communication devices (129) of the interventionist device (121) mayinclude circuitry, hardware, software, or any combination thereof forcommunicating with another device including, for example, transmittingand receiving data. Examples of communication devices may include butare not limited to wireless and wired devices, such as radio frequency(RF) transceivers, Bluetooth transceivers, ethernet controllers,cellular controllers, and wi-fi controllers. In the example of FIG. 1,the connection between the interventionist device (121) and the network(130) is illustrated as a wireless connection. However, readers of skillin the art will realize that the interventionist device (121) may becoupled to the network (130) or another device via a wired connection.

The memory (123) of the interventionist device (121) includes operationinstructions (128) comprising computer program instructions that whenexecuted by the processor (122) cause the processor to control theoperation devices (124), the input devices (125), and the output devices(126).

In the example of FIG. 1, the memory (123) of the interventionist device(121) also includes an interventionist controller (127) that includescomputer program instructions for facilitating human intervention in anautonomous device. Specifically, the interventionist controller (127)includes computer program instructions that when executed by theprocessor (122) cause the processor (122) to: receive a request from theinterventionist service server (101); interface with a humaninterventionist operator including receiving input; generate a responsebased on the received input; and transmit the generated response to theinterventionist service server (101).

The interventionist service server (101) may be any type of computer,device, or machine configured for communicating with autonomous devicesand interventionist devices and facilitating human intervention in anautonomous device according to embodiments of the present disclosure. Inthe example of FIG. 1, the interventionist service server (101) includesa processor (102) coupled to a memory (103), communication devices(109), operation devices (104), input devices (105), and output devices(106). Operation devices may include circuitry, hardware, software, orany combination thereof for performing the various operations of theinterventionist service server.

The input devices (105) may include circuitry, hardware, software, orany combination thereof for capturing, receiving, or otherwise inputtinginformation into the interventionist service server. Examples of inputdevices include but are not limited to cameras, microphones, keyboards,keypads, touchpads, light sensors, radar, positioning circuitry,pressure sensors, fluid sensors, temperature sensors, accelerometers,wheel speed sensors, gyroscopes, altimeters, and many others as willoccur to those of skill in the art. The input devices (105) are coupledto the processor (102), such that data from the input devices may beavailable for use by instructions executed by the processor.

The output devices (106) may include circuitry, hardware, software, orany combination thereof for presenting, displaying, relaying, orotherwise outputting information from the interventionist serviceserver. Examples of output devices include but are not limited todisplay screens, speakers, tactile feedback devices, and many others aswill occur to those of skill in the art. The output devices (106) arecoupled to the processor (102), such that data from instructionsexecuted by the processor may be related to the output devices.

The communication devices (109) of the interventionist service server(101) may include circuitry, hardware, software, or any combinationthereof for communicating with another device including for exampletransmitting and receiving data. Examples of communication devices mayinclude but are not limited to wireless and wired devices, such as radiofrequency (RF) transceivers, Bluetooth transceivers, ethernetcontrollers, cellular controllers, and wi-fi controllers. In the exampleof FIG. 1, the connection between the interventionist service server(101) and the network (130) is illustrated as a wireless connection.However, readers of skill in the art will realize that theinterventionist service server (101) may be coupled to the network (130)or another device via a wired connection.

The memory (103) of the interventionist service server (101) includesoperation instructions (108) comprising computer program instructionsthat when executed by the processor (102) cause the interventionistservice server to control the operation devices (104), the input devices(105), and the output devices (106).

In the example of FIG. 1, the memory (103) of the interventionistservice server (101) also includes a service controller (107) thatincludes computer program instructions for facilitating humanintervention in an autonomous device. Specifically, the servicecontroller (107) includes computer program instructions that whenexecuted by the processor (102) cause the processor (112) to: receive arequest from an autonomous device; determine whether an Autonomy/AImodule of the service controller can perform the request autonomously;in response to determining that the Autonomy/AI module can perform therequest autonomously, provide the request to the Autonomy/AI module; andgenerate, by the Autonomy/AI module, an AI response.

The service controller (107) further includes computer programinstructions that when executed by the processor (102) cause theprocessor (112) to: in response to determining that the Autonomy/AImodule cannot perform the request autonomously, provide the request toan interventionist module of the service controller; select, by theinterventionist module, one or more human interventionists to respond tothe request; transmit, by the interventionist module, the request to oneor more interventionist devices associated with the selected one or morehuman interventionists; and receive one or more interventionistresponses from the one or more interventionist devices.

In the example of FIG. 1, the service controller (107) also includescomputer program instructions that when executed by the processor (102)cause the processor (112) to: determine whether the service controlleris sufficiently confident in the generated response (i.e., AI responseor interventionist response); in response to determining that theservice controller is not sufficiently confident in the generatedresponse, transmit the response to the interventionist module; and inresponse determining that the service controller is sufficientlyconfident in the generated response, transmit the generated response tothe autonomous device.

In a particular embodiment, the service controller (107) is configuredfor receiving a request from an autonomous device; determining whetherthe service controller can perform the request autonomously; in responseto determining that the service controller cannot perform the requestautonomously, selecting from a plurality of human interventionists, afirst set of human interventionists to respond to the request;transmitting the request to a first set of interventionist devicesassociated with the selected first set of human interventionists; andreceiving, from the first set of interventionist devices, a first set ofinterventionist responses to the transmitted request.

In a particular embodiment, the service controller (107) is configuredto select from a first plurality of human interventionists a first setof human interventionists to respond to a request associated with anautonomous device; transmit the request to a first set ofinterventionist devices, each interventionist device of the first set ofinterventionist devices associated with a particular humaninterventionist in the first set of human interventionists; andreceivefrom the first set of interventionist devices, a first set ofinterventionist responses to the request.

In a particular embodiment, the service controller is configured to foreach human interventionist of a plurality of human interventionists:transmit a test request to a particular interventionist deviceassociated with a particular human interventionist; receive from theparticular interventionist device, a test response to the transmittedtest request; and based on the test response, generate a performancemetric of the human interventionist; and based on the performancemetrics of the human interventionists of the plurality of humaninterventionists, determine a rating for each human interventionist ofthe plurality of human interventionists.

The interventionist service server (101), the autonomous device (111),and the interventionist device (121) of FIG. 1 are intended to providerespective examples of an interventionist service server, an autonomousdevice, and an interventionist device and are not intended to limit theconfiguration or components of an interventionist service server, anautonomous device, or the interventionist device that may be used insystems for facilitating human intervention in an autonomous deviceaccording to embodiments of the present disclosure. For example, in aparticular embodiment, the service controller (107) may be implementedin the autonomous device (111). In another embodiment, a plurality ofinterventionist devices may be coupled to the service controller.

The network (130) may comprise one or more Local Area Networks (LANs),Wide Area Networks (WANs), cellular networks, satellite networks,internets, intranets, or other networks and combinations thereof. Thenetwork (130) may comprise one or more wired connections, wirelessconnections, or combinations thereof.

The arrangement of servers and other devices making up the exemplarysystem illustrated in FIG. 1 are for explanation, not for limitation.Data processing systems useful according to various embodiments of thepresent invention may include additional servers, routers, otherdevices, and peer-to-peer architectures, not shown in FIG. 1, as willoccur to those of skill in the art. Networks in such data processingsystems may support many data communications protocols, including forexample TCP (Transmission Control Protocol), IP (Internet Protocol),HTTP (HyperText Transfer Protocol), and others as will occur to those ofskill in the art. Various embodiments of the present invention may beimplemented on a variety of hardware platforms in addition to thoseillustrated in FIG. 1.

For further explanation, FIG. 2 sets forth a flowchart to illustrate animplementation of a method for facilitating human intervention in anautonomous device according to embodiments of the present disclosure.The method of FIG. 2 includes receiving, by a service controller (107)of an interventionist service server (101), a request from an autonomousdevice (111).

In a particular embodiment, the service controller may provide to auser/client of the service, a visual programming interface, where auser/client may quickly program/assemble a Request-type. In thisexample, the user/client may be offered a set of interface types, whichthey can easily mix and match to build the request that is specific totheir task. For instance, a classic robotic assembly task consists ofinserting a peg in a hole. This task requires a specific sequence ofinterface modalities (e.g. different modalities all on a tablet). First,‘locate the peg’ (where the person draws a ‘loop’ around the peg with athink paint-brush style pencil tool). Second, ‘locate the hole’ (wherethe person locates the hole). This is a sequence of a ‘pencil’ tool,followed by a pointer-tool, both associated with a written instruction,e.g. ‘locate the hole’. Using these two pieces of info, the robot cancomplete the task autonomously.

In a particular embodiment, the autonomous device connects to anapplication programming interface (API) of the interventionist serviceserver. The interventionist service server and the autonomous device mayexchange data, such as coordinates. For example, the programmingabstraction that is presented to customers/clients may includecoordinates=getSingleLocation(object_type, image) [object_type, imageare inputs from the autonomous device and sent to the humaninterventionist, coordinates may be what's returned].

An autonomous device operating in an autonomous mode, may send a requestto an interventionist service server for guidance on performing a nextstep. For example, the autonomous device may be traveling along a pathand encounter an obstacle that interrupts the autonomous operation ofthe autonomous device. In this example, the autonomous device may send arequest to the interventionist service server asking for guidance.

The method of FIG. 2 includes in response to receiving the request,determining, by the service controller, whether an autonomy/AI module(202) of the service controller can perform the request autonomously. Inresponse to determining that the autonomy/AI module (202) can performthe request autonomously, the method of FIG. 2 includes the servicecontroller providing the request to the autonomy/AI module (202) forprocessing. In response to providing the autonomy/AI module (202) therequest, the autonomy/AI module generates an autonomous/AI response. Theautonomy/AI module may also be configured to perform generaltraining/learning to adapt future responses based on the responsesgenerated by the autonomy/AI module and the responses generated by thehuman interventionist. In a particular embodiment, the priority orweight applied to the generated responses in the learning process may bebased on various factors including but not limited to the qualificationsof the selected human interventionist; the type of the interventionistdevice; and the urgency of the request.

In response to determining that the autonomy/AI module (202) cannotperform the request autonomously, the method of FIG. 2 includes theservice controller providing the request to an interventionist module(204). The method of FIG. 2 also includes the interventionist module(204) in response to receiving the request, selecting one or more humaninterventionists to process the request. Selecting one or more humaninterventionists may be carried out by selecting, from a plurality ofhuman interventionists having associated interventionist devices, a setof human interventionists based on the qualifications of the humaninterventionists; costs of using the human interventionists;availability of the human interventionists; urgency of the request; typeof interventionist devices associated with the human interventionists;and quality, speed, bandwidth of the connection to the variousinterventionist device. In a particular embodiment, the servicecontroller is configured to receive an updated request. In this case,the service controller's interventionist module may change a selectionof the human interventionist based on the data in the updated request.

In a particular embodiment, human interventionists arebenchmarked/ranked against other humans and/or the autonomy/AI module,with respect to different request types and interfaces. The benchmarkingor rating information may be used to match higher-priority requests (interms of speed or accuracy) to higher-performing interventionists (interms of speed or accuracy). Similarly, autonomous devices of clientswith a higher budget may be coupled to higher-performinginterventionists.

In order to accomplish low-latency (short turnaround) humanintervention, the service controller may utilize statistical methods(based on historical data and extrapolation from which) to a) predictperson's response time (by task/interface type) and b) connectivity(latency/bandwidth) of the internet connection between the autonomousdevice and the human interventionist. Combining a and b, the servicecontroller may optimize for rank latency. Requests may be ranked byurgency and assigned to a human interventionist accordingly.

In order to judge whether a human interventionist is readily availableto respond to a request, the service controller may read the sensors ofthe interventionist device associated with the human interventionist andassess if the interventionist device is actively being used oraccessible by the interventionist (e.g., in the interventionist's hand).Alternatively, the service controller can determine whether the personis online by having him/her operate on simulated data.

The method of FIG. 2 includes the interventionist module (204)transmitting the request to an interventionist device (121) associatedwith the selected human interventionist. In response to receiving therequest, the interventionist device presents the request to the humaninterventionist. The method of FIG. 2 also includes the interventionistdevice receiving input from the human interventionist in response torepresenting the request. In the method of FIG. 2, the interventionistdevice generates and provides to the service controller, based on thereceived input, an interventionist response to the request. For example,in the case where an autonomous device is traveling along a path and isstuck when it encounters an obstacle, a human interventionist may usethe interface of the interventionist device to provide a solution. Inthis example, the solution may be providing coordinates for travelingaround the obstacle.

The method of FIG. 2 includes the service controller (107) determiningwhether the service controller has sufficient confidence in a generatedresponse (i.e., the autonomous/AI response and the interventionistresponse). Certain interventions (perhaps for human-safety criticaltasks) may require high accuracy/confidence. In order to increaseconfidence, the service controller may distribute a single request tomultiple human interventionists. Using statistical methods, the servicecontroller may return intervention data with higher confidence, togetherwith a statistical metric for this confidence (e.g. with 95% confidence,the true object location lies within the selected circle/sphere).

In response to determining that the service controller (107) does nothave sufficient confidence in the generated response, the servicecontroller sends an instruction to the interventionist module (204) toselect one or more human interventionists. If the interventionist modulehad selected a first set of interventionists to respond to the request,the interventionist module may select another set of interventionists torespond to the request.

In response to determining that the service controller (107) does havesufficient confidence in the generated response, the service controllertransmits the generated response (i.e., the autonomous/AI response orthe interventionist response) to the autonomous device (111).

The method of FIG. 2 also includes the autonomous device (111) receivingthe generated response. In response to receiving the generated response,the autonomous device (111) performs an operation in accordance with thegenerated response. For example, a human interventionist may generate aresponse that points out the landing spot of an aerial drone by clickingon a table interventionist device. As another example, a shopping-islescanning robot issues pictures to a human interventionist via theinterventionist service server and the interventionist visually assessesfill-states (e.g. 60% full) through an interface on a smart phoneinterventionist device. As another example, a human operator, through agame pad interventionist device interface, frees a ground delivery dronethat is stuck at an obstacle.

For further explanation, FIG. 3 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 3 includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303). A service controller is circuitry, hardware, software, orsome combination thereof for facilitating human intervention. In aparticular embodiment, the service controller (301) is an example of theservice controller (107) of FIG. 1. As explained above, a servicecontroller may be located on a server or device that is remote from theautonomous device or alternatively, may be located within the autonomousdevice. A request may be a message from the autonomous device thatspecifies information regarding the autonomous device. In a particularembodiment, a request is generated by the service controller based oninformation received from the autonomous device. Examples of the typesof information that the request may include but are not limited to: anoperational status of the autonomous device (e.g., on/off; stuckmessage; error message; alarm codes/messages; damaged equipment; etc.),an instruction/routine/process indication (e.g., right movement loop), alast step/operation/task/behavior performed (e.g., turned left; shutdownrotor), a plurality of decision choices (e.g., menu choices—identifyobjects as (a), (b), or (c)), a type of behavior/task that is requiredto perform (e.g., provide coordinates; guide robot movement), urgencyindication (e.g. respond to five minutes), criteria for selecting ahuman interventionist (e.g., need a doctor; need a mechanic; need alevel 8 technician; need lowest cost, available technician that can dolevel 7 operation on robot); and others as will occur to those of skillin the art after reading the subject disclosure.

Selecting (302) from a first plurality of human interventionists, by aservice controller (301), a first set of human interventionists torespond to a request associated with an autonomous device (303) may becarried out by identifying another set of human interventionists thatbest satisfy the selection criteria that is specified by the servicecontroller or the request. A set of human interventionists may includeone or more human interventionists. A human interventionist is a humanperson that operates an interventionist device. As will be explainedbelow, the selection criteria may include one or more metrics,conditions, or requirements. Examples of the type of data in a selectioncriteria may include but are not limited to: requirements of therequest, such as urgency, priority, difficulty, expense range; and theprofiles of the human interventionists which may include information,such as type of interventionist devices associated with theinterventionists and ratings of the interventionists according tovarious metrics, such as performance metrics, qualification, expense,and availability.

The method of FIG. 3 also includes transmitting (304), by the servicecontroller (301), the request to a first set (305) of interventionistdevices, where each interventionist device of the first set ofinterventionist devices is associated with a particular humaninterventionist in the first set of human interventionists. Transmitting(304), by the service controller (301), the request to a first set (305)of interventionist devices may be carried out by modifying the requestby reformatting, adding data, or removing data from the request; andsending the request via a wireless or wired communication network orconnection to the interventionist device. For example, the request maybe transmitted via an API, as an email, a text message, a link to anetwork location, and any other electronic means of relaying informationfrom one device to another device.

The method of FIG. 3 also includes receiving (306) from the first set(305) of interventionist devices, by the service controller (301), afirst set of interventionist responses to the request. Receiving (306)from the first set (305) of interventionist devices, by the servicecontroller (301), a first set of interventionist responses to therequest may be carried out by receiving data via a wireless or wiredcommunication network or connection to the interventionist device.

As explained above, an autonomous device may be a device that performssome behaviors or tasks in an autonomous manner, without requiring humancontrol or direction. While the autonomous device may perform anoperation in an autonomous manner, the autonomous device may determinethat the instructions/routines/processes of the autonomous device areincapable of identifying a next task to perform with a required degreeof certainty or confidence. In this example, the autonomous device maytransmit a request for help from a service controller (e.g., the servicecontroller (107) of the interventionist service server (101) of FIG. 1).The service controller may include instructions/routines/processes thatcan identify autonomously a next task to perform with the requireddegree of certainty or confidence. In some situations, the servicecontroller may determine that the instructions/routines/processes cannotautonomously identify the next task to perform. In response todetermining that the service controller cannot identify the next task toperform, the service controller may select a human interventionist forperforming the request of the autonomous device.

In a particular embodiment, the service controller may not include an AIor autonomous module with instructions/routines/processes forautonomously identifying the next task to perform. Without an AI orautonomous module to identify the next task to perform, the servicecontroller may be configured to select one or more humaninterventionists for performing the request of the autonomous device.

For further explanation, FIG. 4 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 4 is similar to the method of FIG. 3 in that the method of FIG. 4also includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

However, unlike the method of FIG. 3, the method of FIG. 4 includesbased on the first set of interventionist responses, generating (402),by the service controller (301), a service response to the request.Generating (402) based on the first set of interventionist responses, bythe service controller (301), a service response to the request may becarried out by using the first set of responses to create the serviceresponse; averaging the responses; and modifying the responses byreformatting, adding data, or removing data from the response. Forexample, a set of twelve responses may include ten responses indicatinga first selection and two responses indicating a second selection. Inthis example, the service controller may use the majority response ofthe first selection as the service response.

In addition, the method of FIG. 4 also includes transmitting (404), bythe service controller (301), the service response to the autonomousdevice (303). Transmitting (404), by the service controller (301), theservice response to the autonomous device (303) may be carried out bysending the response via a wireless or wired communication network orconnection to the autonomous device. For example, the request may betransmitted via an API, as an email, a text message, a link to a networklocation, and any other electronic means of relaying information fromone device to another device.

For further explanation, FIG. 5A sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 5A is similar to the method of FIG. 3 in that the method of FIG. 5Aalso includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

However, unlike the method of FIG. 3, the method of FIG. 5A includesdetermining (502), by the service controller (301), whether the servicecontroller (301) is sufficiently confident in the first set ofinterventionist responses. Determining (502), by the service controller(301), whether the service controller (301) is sufficiently confident inthe first set of interventionist responses may be carried out bydetermining the confidence threshold requirement associated with therequest; and determining whether the first set of human interventionistresponses exceeds the confidence threshold. For example, certaininterventions (perhaps for human-safety critical tasks) may require highaccuracy/confidence. In order to increase confidence, the servicecontroller may distribute a single request to multiple humaninterventionists. Using statistical methods, the service controller mayreturn intervention data with higher confidence, together with astatistical metric for this confidence (e.g. with 95% confidence, thetrue object location lies within the selected circle/sphere). However,the returned set of responses may not satisfy the confidence threshold.For example, a confidence threshold may require all responses in the setof responses to indicate the same selection/value/data. Continuing withthis example, if two out of twelve interventionist responses indicate adifferent response, the confidence threshold may be exceeded.

The method of FIG. 5A also includes in response to determining that theservice controller (301) is sufficiently confident in the first set ofinterventionist responses, generating (504), by the service controller(301), a service response to the request. Generating (504), by theservice controller (301), a service response to the request may becarried out by using the first set of responses to create the serviceresponse, averaging the responses, and modifying the responses byreformatting, adding data, or removing data from the responses to formthe service response.

The method of FIG. 5A also includes in response to determining that theservice controller (301) is sufficiently confident in the first set ofinterventionist responses, transmitting (506), by the service controller(301), the first set of interventionist responses to the autonomousdevice (303). Transmitting (506), by the service controller (301), thefirst set of interventionist responses to the autonomous device (305)may be carried out by modifying the first set of responses byreformatting, adding data, or removing data from the responses; andsending the service response via a wireless or wired communicationnetwork or connection to the autonomous device. For example, the requestmay be transmitted via an API, as an email, a text message, a link to anetwork location, and any other electronic means of relaying informationfrom one device to another device.

The method of FIG. 5A also includes in response to determining that theservice controller (301) is not sufficiently confident in the first setof interventionist responses: selecting (508) from a second set of humaninterventionists, by the service controller (301), a second set of humaninterventionists to respond to the request. In a particular embodiment,the second plurality of human interventionists may include one or moreof the human interventionists of the first plurality of humaninterventionists. However, in a particular embodiment, the secondplurality of human interventionists may include a completely differentcollection of human interventionists. Likewise, the second set of humaninterventionists may be the same as the first set, completely differentfrom the first set, or include some but not all of the humaninterventionists of the first set. Selecting (508), by the servicecontroller (301), a second set of human interventionists to respond tothe request may be carried out by identifying another set of humaninterventionists that best satisfy the selection criteria that isspecified by the service controller or the request. As will be explainedbelow, the selection criteria may include one or more metrics,conditions, or requirements. Examples of the type of input for aselection criteria may include but are not limited to: requirements ofthe request, such as urgency, priority, difficulty, expense range; andthe profiles of the human interventionists, which may includeinformation such as the types of interventionist devices associated withthe interventionists and ratings of the interventionists according tovarious metrics, such as qualification, expense, availability,responsiveness, and accuracy.

The method of FIG. 5A further includes in response to determining thatthe service controller (301) is not sufficiently confident in the firstset of interventionist responses: transmitting (510), by the servicecontroller (301), the request to a second set of interventionist devices(505), each interventionist device of the second set of interventionistdevices associated with a particular human interventionist in the secondset of human interventionists. Transmitting (510), by the servicecontroller (301), the request to a second set of interventionist devices(505) associated with the selected set of second human interventionistsmay be carried out by modifying the request by reformatting, addingdata, or removing data from the request; and sending the response via awireless or wired communication network or connection to theinterventionist device. For example, the request may be transmitted viaan API, as an email, a text message, a link to a network location, andany other electronic means of relaying information from one device toanother device.

In addition, the method of FIG. 5A also includes in response todetermining that the service controller (301) is not sufficientlyconfident in the first interventionist response: receiving (512), by theservice controller (301), a second set of interventionist responses fromthe second set of interventionist devices (505). Receiving (512), by theservice controller (301), a second set of interventionist responses fromthe second set of interventionist devices (505) may be carried out byreceiving data via a wireless or wired communication network orconnection to the interventionist device.

For further explanation, FIG. 5B sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 5B is similar to the method of FIG. 5A in that the method of FIG.5B also includes: selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

As in the method of FIG. 5A, the method of FIG. 5B also includes inresponse to determining that the service controller (301) issufficiently confident in the first set of interventionist responses,transmitting (506), by the service controller (301), the first set ofinterventionist responses to the autonomous device (303); and inresponse to determining that the service controller (301) is notsufficiently confident in the first set of interventionist responses:selecting (508) from a second set of human interventionists, by theservice controller (301), a second set of human interventionists torespond to the request; transmitting (510), by the service controller(301), the request to a second set of interventionist devices (505),each interventionist device of the second set of interventionist devicesassociated with a particular human interventionist in the second set ofhuman interventionists; receiving (512), by the service controller(301), a second set of interventionist responses from the second set ofinterventionist devices (505).

However, unlike the method of FIG. 5A, the method of FIG. 5B includesdetermining (520), by the service controller (301), whether the servicecontroller is sufficiently confident in an aggregate of the first set ofinterventionist responses and the second set of interventionistresponses. Determining (520), by the service controller (301), whetherthe service controller is sufficiently confident in an aggregate of thefirst set of interventionist responses and the second set ofinterventionist responses may be carried out by determining theconfidence threshold requirement associated with the request; anddetermining whether the aggregate of the first set of humaninterventionist responses and the second set of human interventionistresponses exceeds the confidence threshold. For example, certaininterventions (perhaps for human-safety critical tasks) may require highaccuracy/confidence. In order to increase confidence, the servicecontroller may distribute a single request to multiple humaninterventionists. Using statistical methods, the service controller mayreturn intervention data with higher confidence, together with astatistical metric for this confidence (e.g. with 95% confidence, thetrue object location lies within the selected circle/sphere). However,the returned set of responses may not satisfy the confidence threshold.For example, a confidence threshold may require all responses in the setof responses to indicate the same selection/value/data. Continuing withthis example, if two out of twelve interventionist responses indicate adifferent response, the confidence threshold may be exceeded.

The method of FIG. 5B also includes in response to determining that theservice controller is sufficiently confident in the aggregate of thefirst set of interventionist responses and the second set ofinterventionist responses: based on the aggregate of the first set ofinterventionist responses and the second set of interventionistresponses, generating (522), by the service controller (301), a serviceresponse to the request. Generating (522), by the service controller(301), a service response to the request may be carried out by using thefirst set of responses and the second set of responses to create theservice response, averaging the responses, and modifying the responsesby reformatting, adding data, or removing data from the responses toform the service response. For example, the most common response may beselected as the basis for the service response. In another example, theaverage of the responses may be selected as the basis for the serviceresponse.

In addition, the method of FIG. 5B also includes in response todetermining that the service controller is sufficiently confident in theaggregate of the first set of interventionist responses and the secondset of interventionist responses: transmitting (524), by the servicecontroller (301), the service response to the autonomous device.Transmitting (524), by the service controller (301), the serviceresponse to the autonomous device may be carried out by modifying thefirst set and the second set of responses by reformatting, adding data,or removing data from the responses; and sending the service responsevia a wireless or wired communication network or connection to theautonomous device. For example, the request may be transmitted via anAPI, as an email, a text message, a link to a network location, and anyother electronic means of relaying information from one device toanother device.

The method of FIG. 5B also includes in response to determining that theservice controller is not sufficiently confident in the aggregate of thefirst set of interventionist responses and the second set ofinterventionist responses: selecting (526) from a third plurality ofhuman interventionists, by the service controller (301), a third set ofhuman interventionists to respond to the request. In a particularembodiment, the third plurality of human interventionists may includeone or more of the human interventionists of the first/second pluralityof human interventionists. However, in a particular embodiment, thethird plurality of human interventionists may include a completelydifferent collection of human interventionists. Likewise, the third setof human interventionists may be the same as the first/second set,completely different from the first/second set, or include some but notall of the human interventionists of the first/second set. Selecting(526) from a third plurality of human interventionists, by the servicecontroller (301), a third set of human interventionists to respond tothe request may be carried out by identifying another set of humaninterventionists that best satisfy the selection criteria that isspecified by the service controller or the request. As will be explainedbelow, the selection criteria may include one or more metrics,conditions, or requirements. Examples of the type of input for aselection criteria may include but are not limited to: requirements ofthe request, such as urgency, priority, difficulty, expense range; andthe profiles of the human interventionists, which may includeinformation such as the types of interventionist devices associated withthe interventionists and ratings of the interventionists according tovarious metrics, such as qualification, expense, availability,responsiveness, and accuracy.

In addition, the method of FIG. 5B also includes in response todetermining that the service controller is not sufficiently confident inthe autonomous response: transmitting (528), by the service controller(301), the request to a third set of interventionist devices (550), eachinterventionist device of the third set of interventionist devicesassociated with a particular human interventionist in the third set ofhuman interventionists. Transmitting (528), by the service controller(301), the request to a third set of interventionist devices, eachinterventionist device of the third set of interventionist devicesassociated with a particular human interventionist in the third set ofhuman interventionists may be carried out by modifying the request byreformatting, adding data, or removing data from the request; andsending the response via a wireless or wired communication network orconnection to the interventionist device. For example, the request maybe transmitted via an API, as an email, a text message, a link to anetwork location, and any other electronic means of relaying informationfrom one device to another device.

Finally, the method of FIG. 5B also includes in response to determiningthat the service controller is not sufficiently confident in theautonomous response: receiving (530), by the service controller (301), athird set of interventionist responses from the third set ofinterventionist devices. Receiving (530), by the service controller(301), a third set of interventionist responses from the third set ofinterventionist devices may be carried out by receiving data via awireless or wired communication network or connection to theinterventionist device.

For further explanation, FIG. 6 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 6 is similar to the method of FIG. 3 in that the method of FIG. 6also includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

However, unlike the method of FIG. 3, the method of FIG. 6 includesdetermining (602), by the service controller (301), whether the servicecontroller (301) can perform the request autonomously. Performing therequest autonomously may include deciding a particular action to performbased only on instructions executing on the autonomous device. That is,in this particular example, performing an action autonomously may becarried out without receiving input from a human or another device.Determining (602), by the service controller (301), whether the servicecontroller (301) can perform the request autonomously may be carried outby determining whether the service controller has a historical record ofsuccessfully performing this type of request; and determining whetherthe priority/criterion of the request allows for the service controllerto perform this type of request.

In the method of FIG. 6, selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303) includes selecting (604) from the first plurality of humaninterventionists, by the service controller (301), the first set ofhuman interventionists to respond to the request in response todetermining that the service controller (301) cannot perform the requestautonomously. Selecting (604) from the first plurality of humaninterventionists, by the service controller (301), the first set ofhuman interventionists to respond to the request in response todetermining that the service controller (301) cannot perform the requestautonomously may be carried out by identifying the one or more humaninterventionists that best satisfy a selection criterion/criteria thatis specified by the service controller or the request. As will beexplained below, the selection criteria may include one or more metrics,conditions, or requirements. Examples of the type of data in a selectioncriteria may include but are not limited to: requirements of therequest, such as urgency, priority, difficulty, expense range; and theprofiles of the human interventionists which may include information,such as type of interventionist devices associated with theinterventionists and ratings of the interventionists according tovarious metrics, such as qualification, expense, availability,responsiveness, and accuracy.

In the method of FIG. 6, transmitting (304), by the service controller(301), the request to a first set (305) of interventionist devicesincludes transmitting (606), by the service controller (301), therequest to the first set of interventionist devices in response todetermining that the service controller (301) cannot perform the requestautonomously. Transmitting (606), by the service controller (301), therequest to the first set of interventionist devices in response todetermining that the service controller (301) cannot perform the requestautonomously may be carried out by modifying the request byreformatting, adding data, or removing data from the request; andsending the request via a wireless or wired communication network orconnection to the interventionist device. For example, the request maybe transmitted via an API, as an email, a text message, a link to anetwork location, and any other electronic means of relaying informationfrom one device to another device.

In the method of FIG. 6, receiving (306) from the first set (305) ofinterventionist devices, by the service controller (301), a first set ofinterventionist responses to the request includes receiving (608) fromthe first set of interventionist devices, by the service controller(301), the first set of interventionist responses to the request inresponse to determining that the service controller (301) cannot performthe request autonomously. Receiving (608) from the first set ofinterventionist devices, by the service controller (301), the first setof interventionist responses to the request in response to determiningthat the service controller (301) cannot perform the requestautonomously may be carried out by receiving data via a wireless orwired communication network or connection to the interventionist device.

For further explanation, FIG. 7A sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 7A is similar to the method of FIG. 6 in that the method of FIG. 7Aalso includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

Like the method of FIG. 6, the method of FIG. 7A also includesdetermining (602), by the service controller (301), whether the servicecontroller (301) can perform the request autonomously; selecting (604)from the first plurality of human interventionists, by the servicecontroller (301), the first set of human interventionists to respond tothe request is in response to determining that the service controller(301) cannot perform the request autonomously; transmitting (606), bythe service controller (301), the request to the first set ofinterventionist devices is in response to determining that the servicecontroller (301) cannot perform the request autonomously; receiving(608) from the first set of interventionist devices, by the servicecontroller (301), the first set of interventionist responses to therequest is in response to determining that the service controller (301)cannot perform the request autonomously.

However, unlike the method of FIG. 6, the method FIG. 7A includes inresponse to determining that the service controller (301) can performthe request autonomously, generating (702), by the service controller(301), an autonomous response. Generating (702), by the servicecontroller (301), an autonomous response in response to determining thatthe service controller (301) can perform the request autonomously may becarried out by using autonomous instructions/processes/routines todecide a particular action to perform based only on instructionsexecuting on the service controller. That is, in this particularexample, performing an action autonomously may be carried out withoutreceiving input from a human or another device. Deciding a particularaction using autonomous instructions/processes/routines may be carriedout by using historical decisions from human interventionists, computersimulations, or data from other sources; usinginstructions/routines/processes to identify patterns in past decisions;calculating probabilities of various solutions producing the correctresult, and selecting the highest probability solution.

For further explanation, FIG. 7B sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 7B is similar to the method of FIG. 7A in that the method of FIG.7B also includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

Like the method of FIG. 7A, the method of FIG. 7B also includesdetermining (602), by the service controller (301), whether the servicecontroller (301) can perform the request autonomously; selecting (604)from the first plurality of human interventionists, by the servicecontroller (301), the first set of human interventionists to respond tothe request is in response to determining that the service controller(301) cannot perform the request autonomously; transmitting (606), bythe service controller (301), the request to the first set ofinterventionist devices is in response to determining that the servicecontroller (301) cannot perform the request autonomously; receiving(608) from the first set of interventionist devices, by the servicecontroller (301), the first set of interventionist responses to therequest is in response to determining that the service controller (301)cannot perform the request autonomously.

Furthermore, as in the method of FIG. 7A, the method FIG. 7B includes inresponse to determining that the service controller (301) can performthe request autonomously, generating (702), by the service controller(301), an autonomous response.

However, unlike the method of FIG. 7A, the method of FIG. 7B includesbased on the autonomous response, generating (720), by the servicecontroller (301), a service response to the request. Generating (720),by the service controller (301), a service response to the request basedon the autonomous response may be carried out by using the autonomousresponse to create the service response, and modifying the autonomousresponse by reformatting, adding data, or removing data from theautonomous response.

The method of FIG. 7B also includes transmitting (722), by the servicecontroller (301), the service response to the autonomous device.Transmitting (722), by the service controller (301), the serviceresponse to the autonomous device may be carried out by sending theresponse via a wireless or wired communication network or connection tothe autonomous device. For example, the request may be transmitted viaan API, as an email, a text message, a link to a network location, andany other electronic means of relaying information from one device toanother device.

For further explanation, FIG. 8 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 8 is similar to the method of FIG. 3 in that the method of FIG. 8also includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

However, the method of FIG. 8 includes determining (802), by the servicecontroller (301), whether the service controller can perform the requestautonomously. Performing the request autonomously may include deciding aparticular action to perform based only on instructions executing on theautonomous device. That is, in this particular example, performing anaction autonomously may be carried out without receiving input from ahuman or another device. Determining (802), by the service controller(301), whether the service controller can perform the requestautonomously may be carried out by determining whether the servicecontroller has a historical record of successfully performing this typeof request; and determining whether the priority/criterion of therequest allows for the service controller to perform this type ofrequest.

The method of FIG. 8 includes in response to determining that theservice controller (301) can perform the request autonomously,generating (804), by the service controller (301), an autonomousresponse. Generating (804), by the service controller (301), anautonomous response in response to determining that the servicecontroller (301) can perform the request autonomously may be carried outby using autonomous instructions/processes/routines to decide aparticular action to perform based only on instructions executing on theservice controller. That is, in this particular example, performing anaction autonomously may be carried out without receiving input from ahuman or another device. Deciding a particular action using autonomousinstructions/processes/routines may be carried out by using historicaldecisions from human interventionists, computer simulations, or datafrom other sources; using instructions/routines/processes to identifypatterns in past decisions; calculating probabilities of varioussolutions producing the correct result, and selecting the highestprobability solution.

In addition, the method of FIG. 8 includes determining (806), by theservice controller (301), whether the service controller is sufficientlyconfident in the autonomous response. Determining (806), by the servicecontroller (301), whether the service controller is sufficientlyconfident in the autonomous response may be carried out by determiningthe confidence threshold requirement associated with the request; anddetermining whether a single autonomous response exceeds the confidencethreshold. For example, certain interventions (perhaps for human-safetycritical tasks) may require high accuracy/confidence. As will beexplained below, in order to increase confidence, the service controllermay distribute a single request to multiple human interventionists.Using statistical methods, the service controller may returnintervention data with higher confidence, together with a statisticalmetric for this confidence (e.g. with 95% confidence, the true objectlocation lies within the selected circle/sphere).

The method of FIG. 8 includes in response to determining that theservice controller is sufficiently confident in the autonomous response:based on the autonomous response, generating (808), by the servicecontroller (301), a service response to the request. Generating (808),by the service controller (301), a service response to the request maybe carried out by using the autonomous response to create the serviceresponse, and modifying the autonomous response by reformatting, addingdata, or removing data from the autonomous response.

The method of FIG. 8 includes in response to determining that theservice controller is sufficiently confident in the autonomous response:transmitting (810), by the service controller (301), the serviceresponse to the autonomous device. Transmitting (810), by the servicecontroller (301), the service response to the autonomous device may becarried out by sending the response via a wireless or wiredcommunication network or connection to the autonomous device. Forexample, the request may be transmitted via an API, as an email, a textmessage, a link to a network location, and any other electronic means ofrelaying information from one device to another device

In the method of FIG. 8, selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303) includes in response to determining that that the servicecontroller is not sufficiently confident in the autonomous response,selecting (812) from a first plurality of human interventionists, by aservice controller (301), a first set of human interventionists torespond to a request associated with an autonomous device (303). In themethod of FIG. 8, transmitting (304), by the service controller (301),the request to a first set (305) of interventionist devices includes inresponse to determining that that the service controller is notsufficiently confident in the autonomous response, transmitting (814),by the service controller (301), the request to a first set (305) ofinterventionist devices. In the method of FIG. 8, receiving (306) fromthe first set (305) of interventionist devices, by the servicecontroller (301), a first set of interventionist responses to therequest includes in response to determining that that the servicecontroller is not sufficiently confident in the autonomous response,receiving (816) from the first set (305) of interventionist devices, bythe service controller (301), a first set of interventionist responsesto the request.

For further explanation, FIG. 9 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 9 is similar to the method of FIG. 8 in that the method of FIG. 9also includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

As in the method of FIG. 8, the method of FIG. 9 also includesdetermining (802), by the service controller (301), whether the servicecontroller can perform the request autonomously; in response todetermining that the service controller (301) can perform the requestautonomously, generating (804), by the service controller (301), anautonomous response; determining (806), by the service controller (301),whether the service controller is sufficiently confident in theautonomous response; in response to determining that the servicecontroller is sufficiently confident in the autonomous response: basedon the autonomous response, generating (808), by the service controller(301), a service response to the request; and transmitting (810), by theservice controller (301), the service response to the autonomous device;in response to determining that that the service controller is notsufficiently confident in the autonomous response, selecting (812) froma first plurality of human interventionists, by a service controller(301), a first set of human interventionists to respond to a requestassociated with an autonomous device (303); in response to determiningthat that the service controller is not sufficiently confident in theautonomous response, transmitting (814), by the service controller(301), the request to a first set (305) of interventionist devices; inresponse to determining that that the service controller is notsufficiently confident in the autonomous response, receiving (816) fromthe first set (305) of interventionist devices, by the servicecontroller (301), a first set of interventionist responses to therequest.

However, unlike the method of FIG. 8, the method of FIG. 9 includesdetermining (902), by the service controller (301), whether the servicecontroller (301) is sufficiently confident in an aggregate of theautonomous response and the first set of interventionist responses.Determining (902), by the service controller (301), whether the servicecontroller (301) is sufficiently confident in the aggregate of theautonomous response and the first set of interventionist responses maybe carried out by determining the confidence threshold requirementassociated with the request; and determining whether the aggregate ofthe autonomous response and the first set of interventionist responsesexceeds the confidence threshold.

The method of FIG. 9 includes in response determining that the servicecontroller (301) is sufficiently confident in aggregate of autonomousresponse and the first set of interventionist responses: based on one ofthe first set of interventionist responses and the aggregate of theautonomous response and the first set of interventionist responses,generating (904), by the service controller (301), a service response tothe request. Generating (904), by the service controller (301), aservice response to the request may be carried out by using theaggregate of the autonomous response and the first set of responses tocreate the service response, averaging the responses, and modifying theresponses by reformatting, adding data, or removing data from theresponse.

The method of FIG. 9 also includes in response determining that theservice controller (301) is sufficiently confident in the aggregate ofthe autonomous response and the first set of interventionist responses:transmitting (906), by the service controller (301), the serviceresponse to the autonomous device. Transmitting (906), by the servicecontroller (301), the service response to the autonomous device may becarried out by sending the response via a wireless or wiredcommunication network or connection to the autonomous device.

In addition, the method of FIG. 9 includes in response to determiningthat the service controller (301) is not sufficiently confident in theaggregate of the autonomous response and the first set ofinterventionist responses: selecting (908) from a second plurality ofhuman interventionists, by the service controller (301), a second set ofhuman interventionists to respond to the request. In a particularembodiment, the second plurality of human interventionists may includeone or more of the human interventionists of the first plurality ofhuman interventionists. However, in a particular embodiment, the secondplurality of human interventionists may include a completely differentcollection of human interventionists. Likewise, the second set of humaninterventionists may be the same as the first set, completely differentfrom the first set, or include some but not all of the humaninterventionists of the first set. Selecting (908) from a secondplurality of human interventionists, by the service controller (301), asecond set of human interventionists to respond to the request may becarried out by identifying another set of human interventionists thatbest satisfies selection criteria that is specified by the servicecontroller or the request. As will be explained below, the selectioncriteria may include one or more metrics, conditions, or requirements.Examples of the type of input for a selection criteria may include butare not limited to: requirements of the request, such as urgency,priority, difficulty, expense range; and the profiles of the humaninterventionists which may include information, such as type ofinterventionist devices associated with the interventionists and ratingsof the interventionists according to various metrics, such asqualification, expense, availability, responsiveness, and accuracy.

The method of FIG. 9 includes in response to determining that theservice controller (301) is not sufficiently confident in the aggregateof the autonomous response and the first set of interventionistresponses: transmitting (910), by the service controller (301), therequest to a second set of interventionist devices, each interventionistdevice of the second set of interventionist devices associated with aparticular human interventionist in the second set of humaninterventionists. Transmitting (910), by the service controller (301),the request to a second set of interventionist devices may be carriedout by sending the response via a wireless or wired communicationnetwork or connection to the interventionist device. For example, therequest may be transmitted via an API, as an email, a text message, alink to a network location, and any other electronic means of relayinginformation from one device to another device.

The method of FIG. 9 includes in response to determining that theservice controller (301) is not sufficiently confident in the aggregateof the autonomous response and the first set of interventionistresponses: receiving (912), by the service controller (301), a secondset of interventionist responses from the second set of interventionistdevices. Receiving (912), by the service controller (301), a second setof interventionist responses from the second set of interventionistdevices may be carried out by receiving data via a wireless or wiredcommunication network or connection to the interventionist device.

For further explanation, FIG. 10 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 10 is similar to the method of FIG. 3 in that the method of FIG. 10includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

However, unlike the method of FIG. 3, in the method of FIG. 10,selecting (302) from a first plurality of human interventionists, by aservice controller (301), a first set of human interventionists torespond to a request associated with an autonomous device (303) includesdetermining (1002), for the request, an expense range and selectioncriteria. An expense range may specify an acceptable expense for using ahuman interventionist. Examples of expense range may include but are notlimited to a not-to-exceed amount; a billing rate; and a formula forcalculating an amount. Selection criteria may specify how the humaninterventionist is to be selected using the expense range. Determining(1002), for the request, an expense range and selection criteria may becarried out by retrieving from data within the request, the expenserange and selection criteria; and using an identifier associated withthe request to lookup in a database or data structure, an expense rangeand selection criteria.

In the method of FIG. 10, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes for one or more human interventionistsof the first plurality of human interventionists, determining (1004) aninterventionist expense for each of the one or more humaninterventionists, the interventionist expense indicating the cost ofusing the human interventionist to respond to the request. Determining(1004) an interventionist expense for each of the one or more humaninterventionists may be carried out by retrieving the expense from adata structure that specifies the expense of the human interventionist.In a particular embodiment, determining the expense may be carried outby calculating the expense for a human interventionist based on aspecific time of the request and the fees associated with the humaninterventionist performing at the specific time.

In the method of FIG. 10, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes selecting (1006), from the firstplurality of human interventionists, the first set of humaninterventionists based on the expense range, the selection criteria, andthe interventionist expense of each of the one or more humaninterventionists. Selecting (1006), from the first plurality of humaninterventionists, the first set of human interventionists based on theexpense range, the selection criteria, and the interventionist expenseof each of the one or more human interventionists may be carried out byidentifying the interventionist that best satisfies the selectioncriteria.

For example, a first human interventionist located in China may charge$100 for requests received between 1:30 GMT to 10:30 GMT and $300 forrequests received between 10:30 GMT to 12:30 GMT. Continuing with thisexample, a second human interventionist located in Canada may charge$200 for requests received between 10:00 GMT to 18:00 GMT. In thisexample, if a first request is received at 10:15 GMT and the selectioncriteria requires that the human interventionist having the lowestexpense should be selected, the service controller may select the firsthuman interventionist. However, if a second request having the sameselection criteria is received at 10:40 GMT, the service controller mayselect the second human interventionist.

For further explanation, FIG. 11 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 11 is similar to the method of FIG. 3 in that the method of FIG. 11also includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

However, unlike the method of FIG. 3, in the method of FIG. 11,selecting (302) from a first plurality of human interventionists, by aservice controller (301), a first set of human interventionists torespond to a request associated with an autonomous device (303) includesdetermining (1102), for the request, a difficulty rating and selectioncriteria. A difficulty rating may be an indication of the skillsrequired to respond to the request. Determining (1102), for the request,a difficulty rating and selection criteria may be carried out byretrieving from data within the request, the difficulty rating andselection criteria; using an identifier associated with the request tolookup in a database or data structure, the difficulty rating andselection criteria.

In the method of FIG. 11, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes for one or more human interventionistsof the first plurality of human interventionists, determining (1104) aqualification rating of each of the one or more human interventionists.A qualification rating may be an indication of the skills of the humaninterventionist. Determining (1104) a qualification rating of each ofthe one or more human interventionists may be carried out by retrievingthe qualification rating from a data structure that specifies the ratingof the human interventionist.

In the method of FIG. 11, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes selecting (1106), from the firstplurality of human interventionists, the first set of humaninterventionists based on the difficulty rating, the selection criteria,and the qualification rating of the one or more human interventionists.Selecting (1106), from the first plurality of human interventionists,the first set of human interventionists based on the difficulty rating,the selection criteria, and the qualification rating of the one or morehuman interventionists may be carried out by identifying theinterventionist that best satisfies the selection criteria.

In a particular embodiment, the selection criteria may specify selectingthe human interventionist having the lowest qualification rating thatstill satisfies the difficulty rating of the request. For example, amedical robot may send a request for help closing a suture. In thisexample, the difficult rating may require a qualification ratingassociated with a nurse level practitioner. Continuing with thisexample, based on the criteria, the service controller may select afirst human interventionist having a qualification rating indicating anurse level practitioner over a second human interventionist having aqualification rating indicating a surgeon level practitioner.Alternatively, the selection criteria may require selecting the humanintervention having the highest qualification rating. As explainedabove, the qualification rating and difficulty rating may be used inconjunction with other metrics, such as expense, availability, andperformance metrics to select a human interventionist.

For further explanation, FIG. 12 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 12 is similar to the method of FIG. 3 in that the method of FIG. 12also includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

However, unlike the method of FIG. 3, the method of FIG. 12, selecting(302) from a first plurality of human interventionists, by a servicecontroller (301), a first set of human interventionists to respond to arequest associated with an autonomous device (303) includes for one ormore human interventionists of the first plurality of humaninterventionists, determining (1202), by the service controller (301), aresponsiveness of each of the one or more human interventionists.Determining (1202), by the service controller (301), a responsiveness ofeach of the one or more human interventionists may be carried out byusing historical data indicating past response times of the humaninterventionist; and using test results that indicate recent or currentresponsiveness of the human interventionist. In a particular embodiment,determining (1202), by the service controller (301), a responsiveness ofthe human interventionist may be carried out by periodically testing ahuman interventionist by sending sample requests to the humaninterventionist; measuring the time for the human interventionist torespond; and determining whether the measured time exceeds a threshold.Alternatively, the service controller may measure the signals from theinterventionist device to determine whether the human interventionistsis active on the device. In this example, the service controller maydetermine whether the level of activity exceeds a threshold.

In the method of FIG. 12, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes determining (1204), for the request, anurgency rating and selection criteria. Determining (1204), for therequest, an urgency rating and selection criteria may be carried out byretrieving from data within the request, the urgency rating andselection criteria; and using an identifier associated with the requestto lookup in a database or data structure, an urgency rating andselection criteria.

In the method of FIG. 12, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes selecting (1206), from the firstplurality of human interventionists, the first set of humaninterventionists based on the urgency rating, the selection criteria,and the responsiveness of each of the one or more humaninterventionists. Selecting (1206), from the first plurality of humaninterventionists, the first set of human interventionists based on theurgency rating, the selection criteria, and the responsiveness of eachof the one or more human interventionists may be carried out byidentifying the human interventions that best satisfies the selectioncriteria.

For example, an urgency rating may specify that a response to theparticular request is received within two minutes. In this example, theservice controller may determine that two human interventionists havepredicted latency values that indicate the human interventionist wouldrespond in under two minutes. The service controller may apply othermetrics/criteria to select between the two identified humaninterventionists. For example, the selection criterion may specifyselecting the human interventionist with the highest qualificationrating and the lowest expense that meets an urgency latency threshold.

For further explanation, FIG. 13 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 13 is similar to the method of FIG. 3 also includes selecting (302)from a first plurality of human interventionists, by a servicecontroller (301), a first set of human interventionists to respond to arequest associated with an autonomous device (303); transmitting (304),by the service controller (301), the request to a first set (305) ofinterventionist devices; and receiving (306) from the first set (305) ofinterventionist devices, by the service controller (301), a first set ofinterventionist responses to the request.

However, unlike the method of FIG. 3, in the method of FIG. 13,selecting (302) from a first plurality of human interventionists, by aservice controller (301), a first set of human interventionists torespond to a request associated with an autonomous device (303) includesfor one or more human interventionists of the first plurality of humaninterventionists, determining (1302), by the service controller (301), aperformance metric of each of the one or more human interventionists. Aperformance metric is an indicator of the performance of a humaninterventionist. Examples of performance metrics include but are notlimited to assessments of latency, responsiveness, bandwidth, accuracy,precision, recall, F-1, and repeatability. Determining (1302), by theservice controller (301), a performance metric of the humaninterventionist may be carried out by examining historical performanceof the human interventionists; and periodically testing the humaninterventionist.

In the method of FIG. 13, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes determining (1304), for the request, aperformance requirement and selection criteria. A performancerequirement may be an indication of a threshold number associated with aparticular type of performance metric. The selection criteria mayspecify how the human intervention is to be selected using theperformance requirement. Determining (1304), for the request, aperformance requirement and selection criteria may be carried out byretrieving from data within the request, the performance requirement andselection criteria; and using an identifier associated with the requestto lookup in a database or data structure, the performance requirementand selection criteria.

In the method of FIG. 13, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes selecting (1306), from the firstplurality of human interventionists, the first set of humaninterventionists based on the performance requirement, the selectioncriteria, and the performance metric of each of the one or more humaninterventionists. Selecting (1306), from the first plurality of humaninterventionists, the first set of human interventionists based on theperformance requirement, the selection criteria, and the performancemetric of each of the one or more human interventionists may be carriedout by identifying the interventionist that best satisfies the selectioncriteria.

For example, one performance metric may indicate a score on a test thatrequires a human intervention to identify objects. In this example, theperformance requirement may specify a threshold test score and theselection criteria may specify selecting a human interventionist with ascore above the threshold.

For further explanation, FIG. 14 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 14 is similar to the method of FIG. 3 in that the method of FIG. 14also includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

However, unlike the method of FIG. 3, in the method of FIG. 14,selecting (302) from a first plurality of human interventionists, by aservice controller (301), a first set of human interventionists torespond to a request associated with an autonomous device (303) includesdetermining (1402), by the service controller (301), a confidencerequirement associated with the request. In a particular embodiment,performances of human interventionists may be modeled to estimateaccuracy and consistency of the human interventionists performing thetask. Determining (1402), by the service controller (301), a confidencerequirement associated with the request may be carried out by modelingthe performance of a group of human interventionists to determine theparameters and qualities of the human interventionists that would berequired to perform a particular task with a required level of accuracyand consistency. Examples of modeling methods include but are notlimited to means, medians, regressions, probability distributions,mixture models (e.g., Gaussian), Kernel Density Estimation, andclustering.

In the method of FIG. 14, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes determining (1404) a number of humaninterventionists needed to satisfy the confidence requirement.Determining (1404) a number of human interventionists needed to satisfythe confidence requirement may be carried out by using a model todetermine a minimum number of human interventionists for a set of humaninterventionists to achieve as a group, a desired accuracy andconsistency.

In the method of FIG. 14, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes including (1406) at least thedetermined number of human interventionists within the first set ofhuman interventionists.

For further explanation, FIG. 15 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 15 is similar to the method of FIG. 3 in that the method of FIG. 15also includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

However, unlike the method of FIG. 3, in the method of FIG. 15,selecting (302) from a first plurality of human interventionists, by aservice controller (301), a first set of human interventionists torespond to a request associated with an autonomous device (303) includesdetermining (1502) a type of task to be performed, by a humaninterventionist, for responding to the request. Examples of types oftasks to be performed may include but are not limited to identifying anobject to select; identifying a location; identifying a label for anobject or action; selecting from a plurality of choices, an action ortask; identifying a route; selecting from one or more menus; andproviding a voice or text message. Determining (1502) a type of task tobe performed, by a human interventionist, for responding to the requestmay be carried out by determining a task code within the request; andanalyzing data within the request.

In the method of FIG. 15, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes determining (1504) a type ofinterventionist device for performing the task. Determining (1504) atype of interventionist device for performing the task may be carriedout by matching a task code with an interventionist device code.

In the method of FIG. 15, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes selecting (1506) for the first set ofhuman interventionists, a particular human interventionist having aparticular interventionist device matching the determined type ofinterventionist device. Selecting (1506) for the first set of humaninterventionists, a particular human interventionist having a particularinterventionist device matching the determined type of interventionistdevice may be carried out by identifying a human interventionist havingan associated interventionist device that matches/satisfies the type ofrequired interventionist device.

For example, a request for movement guidance from a medical robot mayrequire a human interventionist to use a specific interventionistdevice, such as a virtual reality glove, to provide the response. Asanother example, a request for landing coordinates from a UAV mayrequire a human interventionist to use a joystick as the interventionistdevice to provide the response.

For further explanation, FIG. 16 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 16 is similar to the method of FIG. 3 in that the method of FIG. 3is similar to the method of FIG. 16 also includes selecting (302) from afirst plurality of human interventionists, by a service controller(301), a first set of human interventionists to respond to a requestassociated with an autonomous device (303); transmitting (304), by theservice controller (301), the request to a first set (305) ofinterventionist devices; and receiving (306) from the first set (305) ofinterventionist devices, by the service controller (301), a first set ofinterventionist responses to the request.

In the method of FIG. 16, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes determining (1602) a type of task to beperformed by a human interventionist, for responding the request.Examples of types of tasks to be performed may include but are notlimited to identifying an object to select; identifying a location;identifying a label for an object or action; selecting from a pluralityof choices, an action or task; identifying a route; selecting from oneor more menus; providing a voice or text message. Determining (1602) atype of task to be performed by a human interventionist, for respondingthe request may be carried out by matching a task code with aninterventionist device code.

In the method of FIG. 16, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes selecting (1604) for the first set ofhuman interventionists, a particular human interventionist having aprofile indicating the capability of performing the type of task to beperformed for responding to the request. Selecting (1604) for the firstset of human interventionists, a particular human interventionist havinga profile indicating the capability of performing the type of task to beperformed for responding to the request may be carried out by examininga profile for an indication that the human interventionist can performthe task or behavior required in the request; and matching the task codewith an interventionist skill code in the interventionist's profile.

For further explanation, FIG. 17 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 17 is similar to the method of FIG. 3 also includes selecting (302)from a first plurality of human interventionists, by a servicecontroller (301), a first set of human interventionists to respond to arequest associated with an autonomous device (303); transmitting (304),by the service controller (301), the request to a first set (305) ofinterventionist devices; and receiving (306) from the first set (305) ofinterventionist devices, by the service controller (301), a first set ofinterventionist responses to the request.

However, unlike the method of FIG. 3, the method of FIG. 17 includesattributing (1702) to an account associated with the autonomous device,by the service controller (1301), the particular expense associated witheach human interventionist in the first set of human interventionists.In a particular embodiment, each human interventionist in the first setof human interventionists has an associated expense for performing aparticular type of request. Attributing (1702) to an account associatedwith the autonomous device, by the service controller (1301), theparticular expense associated with each human interventionist in thefirst set of human interventionists may be carried out by calculatingthe expense of the selected human interventionists and debiting theexpense to an account associated with the interventionist device.

For further explanation, FIG. 18 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure that includesselecting (302) from a first plurality of human interventionists, by aservice controller (301), a first set of human interventionists torespond to a request associated with an autonomous device (303);transmitting (304), by the service controller (301), the request to afirst set (305) of interventionist devices; and receiving (306) from thefirst set (305) of interventionist devices, by the service controller(301), a first set of interventionist responses to the request.

The method of FIG. 18 includes for one or more human interventionists inthe first plurality of human interventionists: transmitting (1802), bythe service controller (1301), a test request to a particularinterventionist device associated with a particular humaninterventionist. Transmitting (1802), by the service controller (1301),a test request to a particular interventionist device associated with aparticular human interventionist may be carried out by sending ahistorical request to the interventionist device; and sending a samplerequest to the interventionist device.

The method of FIG. 18 also includes for one or more humaninterventionists in the first plurality of human interventionists:receiving (1804) from the particular interventionist device, by theservice controller (1301), a test response to the transmitted testrequest. Receiving (1804) from the particular interventionist device, bythe service controller (1301), a test response to the transmitted testrequest may be carried out by receiving data in response to the request.

The method of FIG. 18 includes for one or more human interventionists inthe first plurality of human interventionists: based on the testresponse, generating (1806), by the service controller (1301), aperformance metric of the human interventionist. Generating (1806), bythe service controller (1301), a performance metric of the humaninterventionist may be carried out by comparing the response to ahistorical response; comparing the response to a known or preferredsolution of the request.

In addition, the method of FIG. 18 also includes based on theperformance metrics of each of the one or more human interventionists ofthe first plurality of human interventionists, determining (1808), bythe service controller (1301), a rating for each of the one or morehuman interventionists. Determining (1808), by the service controller(1301), a rating for each of the one or more human interventionists maybe carried out by sorting the results of the comparisons.

In the method of FIG. 18, selecting (302) from a first plurality ofhuman interventionists, by a service controller (301), a first set ofhuman interventionists to respond to a request associated with anautonomous device (303) includes selecting (1810), based on thedetermined ratings, the first set of human interventionists. Selecting(1810), based on the determined ratings, the first set of humaninterventionists may be carried out by identifying and selecting thehuman interventionist with the rating indicating the lowest latency.

For further explanation, FIG. 19A sets forth a flow chart illustratingan exemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 19A is similar to the method of FIG. 18 in that the method of FIG.19A also includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

As in the method of FIG. 18, the method of FIG. 19A also includes foreach of the human interventionists of the first plurality of humaninterventionists: transmitting (1802), by the service controller (1301),a test request to a particular interventionist device associated with aparticular human interventionist; receiving (1804) from the particularinterventionist device, by the service controller (1301), a testresponse to the transmitted test request; and based on the testresponse, generating (1806), by the service controller (1301), aperformance metric of the human interventionist. The method of FIG. 19Aalso includes based on the performance metrics of the humaninterventionists of the first plurality of human interventionists,determining (1808), by the service controller (1301), a rating for eachhuman interventionist of the first plurality of human interventionists;and selecting (1810), based on the determined ratings, the first set ofhuman interventionists.

However, unlike the method FIG. 18, the method of FIG. 19A includestransmitting (1902) the test response to a human interventionistevaluator. A human intervention evaluator may be a supervisor humaninterventionist. Alternatively, the human interventionist evaluator maybe an autonomous module that includes computer program instructions.Transmitting (1902) the test response to a human interventionistevaluator may be carried out by modifying the test response byreformatting, adding data, or removing data from the test response; andsending the test response via a wireless or wired communication networkor connection to the human interventionist evaluator. For example, therequest may be transmitted via an API, as an email, a text message, alink to a network location, and any other electronic means of relayinginformation from one device to another device.

The method of FIG. 19A also includes receiving (1904) from the humaninterventionist evaluator, an evaluation of the test response. Anevaluation of the test response may include a binary decision of whetherthe human interventionist made the correct decision. The evaluation mayalso include a determination or indication of the correct or preferredsolution. Receiving (1904) from the human interventionist evaluator, anevaluation of the test response may be carried out by receiving data viaa wireless or wired communication network or connection to the humaninterventionist evaluator.

For further explanation, FIG. 19B sets forth a flow chart illustratingan exemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 19B is similar to the method of FIG. 3 in that the method of FIG.19B also includes selecting (302) from a first plurality of humaninterventionists, by a service controller (301), a first set of humaninterventionists to respond to a request associated with an autonomousdevice (303); transmitting (304), by the service controller (301), therequest to a first set (305) of interventionist devices; and receiving(306) from the first set (305) of interventionist devices, by theservice controller (301), a first set of interventionist responses tothe request.

However, unlike the method of FIG. 3, in the method of FIG. 19B,selecting (302) from a first plurality of human interventionists, by aservice controller (301), a first set of human interventionists torespond to a request associated with an autonomous device (303) includesdetermining (1920), by the service controller (301), whether aparticular human interventionist is available. Determining (1920), bythe service controller (301), whether a particular human interventionistis available may be carried out by periodically testing a humaninterventionist by sending sample requests to the human interventionist;measuring the time for the human interventionist to respond; anddetermining whether the measured time exceeds a threshold.Alternatively, the service controller may measure the signals from theinterventionist device to determine whether the human interventionistsis active on the device. In this example, the service controller maydetermine whether the level of activity exceeds a threshold.

The method of FIG. 19B also includes in response to determining that theparticular human interventionist is not available, excluding (1922), bythe service controller (301), the particular human interventionist fromthe first set of human interventionists to respond to the request.Excluding (1922), by the service controller (301), the particular humaninterventionist from the first set of human interventionists to respondto the request may be carried out by removing the human interventionistfrom the list of human interventionists that are available to beincluded in the plurality of human interventionists

In addition, the method of FIG. 19B also includes in response todetermining that the particular human interventionist is available,adding (1924), by the service controller (301), the particular humaninterventionist to the first set of human interventionists to respond tothe request. Adding (1924), by the service controller (301), theparticular human interventionist to the first set of humaninterventionists to respond to the request may be carried out by addingthe human interventionist to the list of human interventionists that areavailable to be included in the plurality of human interventionists.

For further explanation, FIG. 20 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 20 includes for each human interventionist of a plurality of humaninterventionists: transmitting (2002), by a service controller (2001), atest request to a particular interventionist device associated with aparticular human interventionist. Transmitting (2002), by a servicecontroller (2001), a test request to a particular interventionist deviceassociated with a particular human interventionist may be carried out bysending a historical request to the interventionist device; and sendinga sample request to the interventionist device.

The method of FIG. 20 includes for each human interventionist of aplurality of human interventionists: receiving (2004) from theparticular interventionist device, by the service controller (2001), atest response to the transmitted test request. Receiving (2004) from theparticular interventionist device, by the service controller (2001), atest response to the transmitted test request may be carried out byreceiving data in response to the request.

In addition, the method of FIG. 20 includes for each humaninterventionist of a plurality of human interventionists: based on thetest response, generating (2006), by the service controller (2001), aperformance metric of the human interventionist. Generating (2006), bythe service controller (2001), a performance metric of the humaninterventionist may be carried out by comparing the response to ahistorical response; and comparing the response to a known or preferredsolution of the request.

The method of FIG. 20 includes based on the performance metrics of thehuman interventionists of the plurality of human interventionists,determining (2008), by the service controller (2001), a rating for eachhuman interventionist of the plurality of human interventionists.Determining (2008), by the service controller (2001), a rating for eachhuman interventionist of the plurality of human interventionists may becarried out by comparing the performance metrics of each humaninterventionists and sorting the results of the comparison.

For further explanation, FIG. 21 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 21 is similar to the method of FIG. 20 in that the method of FIG.21 also includes for each human interventionist of a plurality of humaninterventionists: transmitting (2002), by a service controller (2001), atest request to a particular interventionist device associated with aparticular human interventionist; receiving (2004) from the particularinterventionist device, by the service controller (2001), a testresponse to the transmitted test request; and based on the testresponse, generating (2006), by the service controller (2001), aperformance metric of the human interventionist. The method of FIG. 21also includes based on the performance metrics of the humaninterventionists of the plurality of human interventionists, determining(2008), by the service controller (2001), a rating for each humaninterventionist of the plurality of human interventionists.

In addition, the method of FIG. 21 also includes receiving (2102), bythe service controller (2001), a request from an autonomous device. Arequest may be a message from the autonomous device that specifiesinformation regarding the autonomous device. Examples of the types ofinformation that the request may include but are not limited to: anoperational status of the autonomous device (e.g., on/off; stuckmessage; error message; alarm codes/messages; damaged equipment; etc.),an instruction/routine/process indication (e.g., right movement loop), alast step/operation/task/behavior performed (e.g., turned left; shutdownrotor), a plurality of decision choices (e.g., menu choices—identifyobjects as (a), (b), or (c)), a type of behavior/task that is requiredto perform (e.g., provide coordinates; guide robot movement), urgencyindication (e.g. respond to five minutes), criteria for selecting ahuman interventionist (e.g., need a doctor; need a mechanic; need alevel 8 technician; need lowest cost, available technician that can dolevel 7 operation on robot); and others as will occur to those of skillin the art after reading the subject disclosure. Receiving (2102), bythe service controller (301), a request from an autonomous device may becarried out by receiving via wired or wireless communication, a messagefrom the autonomous device.

The method of FIG. 21 also includes selecting (2104) from the pluralityof human interventionists, by the service controller (2001), a first setof human interventionists to respond to the request. Selecting (2104)from the plurality of human interventionists, by the service controller(2001), a first set of human interventionists to respond to the requestmay be carried out by identifying a set of human interventionists thatbest satisfy the selection criteria that is specified by the servicecontroller or the request.

The method of FIG. 21 also includes transmitting (2106), by the servicecontroller (2001), the request to a first set of interventionistdevices, each interventionist device of the first set of interventionistdevices associated with a particular human interventionist in the firstset of human interventionists. Transmitting (2106), by the servicecontroller (2001), the request to a first set of interventionist devicesmay be carried out by modifying the request by reformatting, addingdata, or removing data from the request; and sending the response via awireless or wired communication network or connection to theinterventionist device. For example, the request may be transmitted viaan API, as an email, a text message, a link to a network location, andany other electronic means of relaying information from one device toanother device.

The method of FIG. 21 also includes receiving (2108) from the first setof interventionist devices, by the service controller (2001), a firstset of interventionist responses to the request. Receiving (2108) fromthe first set of interventionist devices, by the service controller(2001), a first set of interventionist responses to the request may becarried out by receiving data via a wireless or wired communicationnetwork or connection to the interventionist device.

For further explanation, FIG. 22 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 22 includes placing (2202), by an autonomous device (2201), anobject in a stack of one or more objects. Placing (2202), by anautonomous device (2201), an object in a stack of one or more objectsmay be carried out by using an autonomous process to identify an objectto move, determining how to grasp the object, grasping the object usingone or components of the autonomous device, identifying a location toplace the object, moving the object to the location, and releasing theobject.

The method of FIG. 22 also includes in response to placing the object inthe stack, detecting (2204), by the autonomous device (2201),instability in the stack. Detecting (2204), by the autonomous device(2201), instability in the stack in response to placing the object inthe stack may be carried out by receiving sensor input associated withthe object; analyzing the sensor input; comparing the sensor input tohistorical records, and determining, based on the analysis of the sensorinput, that the object or stack is unstable. For example, the autonomousdevice may determine from pictures of the stack that one or more objectsin the stack forms a configuration that historically has shown to beunstable.

In addition, the method of FIG. 22 also includes determining (2206), bythe autonomous device (2201), that correcting the instability requires alevel of capability beyond a threshold associated with the autonomousdevice. Determining (2206), by the autonomous device (2201), thatcorrecting the instability requires a level of capability beyond athreshold associated with the autonomous device may be carried out byanalyzing a historical record to determine whether the identifiedunstable configuration has successfully been stabilized by theautonomous device. For example, the autonomous device may determine thatthe autonomous device has previously attempted unsuccessfully tostabilize this type of stacked configuration.

The method of FIG. 22 also includes in response to determining thatcorrecting the instability requires the level of capability beyond thethreshold associated with the autonomous device, generating (2208), bythe autonomous device (2201), a request for human intervention.Generating (2208), by the autonomous device (2201), a request for humanintervention in response to determining that correcting the instabilityrequires the level of capability beyond the threshold associated withthe autonomous device may be carried out by transmitting to a servicecontroller, a message requesting human intervention in stabilizing thestack.

For further explanation, FIG. 23 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 23 includes grasping (2302), by an autonomous device (2301), anobject with one or more components of the autonomous device. Grasping(2302), by an autonomous device (2301), an object with one or morecomponents of the autonomous device may be carried out by identify anobject to move, determining how to grasp the object, and grasping theobject using one or components of the autonomous device.

The method of FIG. 23 also includes based on sensor input, determining(2304), by the autonomous device (2301), a level of confidence that theobject is securely grasped with the one or more components of theautonomous device (2301). Determining (2304) based on sensor input, bythe autonomous device (2301), a level of confidence that the object issecurely grasped with the one or more components of the autonomousdevice (2301) may be carried out by receiving sensor input (e.g.,tactile sensors, pressure sensors, camera input, etc.) associated withthe object and the components of the autonomous device; analyzing thesensor input; comparing the sensor input to historical records; anddetermining, based on the analysis of the sensor input, that the objectis not securely grasped.

In addition, the method of FIG. 23 also includes determining (2306), bythe autonomous device (2301), that the level does not satisfy aconfidence requirement. Determining (2306), by the autonomous device(2301), that the level does not satisfy a confidence requirement may becarried out by determining based on the sensor input that the object hasa particular probability of coming loose from the grasp of theautonomous device.

The method of FIG. 23 also includes in response to determining that thelevel does not satisfy the confidence requirement, generating (2308), bythe autonomous device (2301), a request for human intervention.Generating (2308), by the autonomous device (2301), a request for humanintervention in response to determining that the level does not satisfythe confidence requirement may be carried out by determining that theprobability of the object coming loose exceeds a threshold associatedwith the confidence requirement.

For further explanation, FIG. 24 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 24 includes receiving (2402), by a service controller (2401), arequest from an autonomous device. Receiving (2402), by a servicecontroller (2401), a request from an autonomous device may be carriedout by receiving via a wireless or wired connection, a message thatspecifies information regarding the autonomous device. Examples of thetypes of information that the request may include but are not limitedto: an operational status of the autonomous device (e.g., on/off; stuckmessage; error message; alarm codes/messages; damaged equipment; etc.),an instruction/routine/process indication (e.g., right movement loop), alast step/operation/task/behavior performed (e.g., turned left; shutdownrotor), a plurality of decision choices (e.g., menu choices—identifyobjects as (a), (b), or (c)), a type of behavior/task that is requiredto perform (e.g., provide coordinates; guide robot movement), urgencyindication (e.g. respond to five minutes), criteria for selecting ahuman interventionist (e.g., need a doctor; need a mechanic; need alevel 8 technician; need lowest cost, available technician that can dolevel 7 operation on robot); and others as will occur to those of skillin the art after reading the subject disclosure.

The method of FIG. 24 also includes generating (2404), by the servicecontroller (2401), a service response to the request, the serviceresponse generated based on least one of: a first set of interventionistresponses received from a first set of interventionist devicesassociated with a first set of human interventionists; and a first setof autonomous responses generated by an autonomous module. In aparticular embodiment, the service controller may have an autonomousmodule that is capable of generating a set of autonomous responses(e.g., one or more responses). The service controller may also send arequest to one or more human interventionist devices and receiveinterventionist responses. Generating (2404), by the service controller(2401), a service response to the request, the service responsegenerated based on least one of: a first set of interventionistresponses received from a first set of interventionist devicesassociated with a first set of human interventionists; and a first setof autonomous responses generated by an autonomous module may be carriedout by using one or more of the first set of autonomous responses andthe first set of interventionist responses to create the serviceresponse, averaging the responses, and modifying the responses byreformatting, adding data, or removing data from the responses.

In addition, the method of FIG. 24 also includes training (2406), by theservice controller (2401), the autonomous module using at least one ofthe first set of interventionist responses and the first set ofautonomous responses. Training (2406), by the service controller (2401),the autonomous module using at least one of the first set ofinterventionist responses and the first set of autonomous responses maybe carried out by determining the most common response among theresponses (autonomous responses and interventionist responses); usingthe determined most common response as the response that the autonomousmodule should provide in response to receiving a similar request. In aparticular embodiment, the service controller may determine the outcomeof the service response to determine whether the service response isused as the response that the autonomous module should provide inresponse to receiving a similar response. In another embodiment, theservice controller may utilize feedback from the autonomous device todetermine whether the service response is used as the response that theautonomous module should provide in response to receiving a similarresponse.

For further explanation, FIG. 25 sets forth a flow chart illustrating anexemplary method for facilitating human intervention in an autonomousdevice according to embodiments of the present disclosure. The method ofFIG. 25 is similar to the method of FIG. 24 in that the method of FIG.25 also includes receiving (2402), by a service controller (2401), arequest from an autonomous device; generating (2404), by the servicecontroller (2401), a service response to the request, the serviceresponse generated based on least one of: a first set of interventionistresponses received from a first set of interventionist devicesassociated with a first set of human interventionists; and a first setof autonomous responses generated by an autonomous module; and training(2406), by the service controller (2401), the autonomous module using atleast one of the first set of interventionist responses and the firstset of autonomous responses.

However, unlike the method of FIG. 24, the method of FIG. 25 includestransmitting (2502), by the service controller (2401), the serviceresponse to the autonomous device. Transmitting (2502), by the servicecontroller (2401), the service response to the autonomous device may becarried out by sending the response via a wireless or wiredcommunication network or connection to the autonomous device.

Exemplary embodiments of the present invention are described largely inthe context of a fully functional computer system for facilitating humanintervention in an autonomous device. Readers of skill in the art willrecognize, however, that the present invention also may be embodied in acomputer program product disposed upon computer readable storage mediafor use with any suitable data processing system. Such computer readablestorage media may be any storage medium for machine-readableinformation, including magnetic media, optical media, or other suitablemedia. Examples of such media include magnetic disks in hard drives ordiskettes, compact disks for optical drives, magnetic tape, and othersas will occur to those of skill in the art. Persons skilled in the artwill immediately recognize that any computer system having suitableprogramming means will be capable of executing the steps of the methodof the invention as embodied in a computer program product. Personsskilled in the art will recognize also that, although some of theexemplary embodiments described in this specification are oriented tosoftware installed and executing on computer hardware, nevertheless,alternative embodiments implemented as firmware or as hardware are wellwithin the scope of the present invention.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Hardware logic, including programmable logic for use with a programmablelogic device (PLD) implementing all or part of the functionalitypreviously described herein, may be designed using traditional manualmethods or may be designed, captured, simulated, or documentedelectronically using various tools, such as Computer Aided Design (CAD)programs, a hardware description language (e.g., VHDL or Verilog), or aPLD programming language. Hardware logic may also be generated by anon-transitory computer readable medium storing instructions that, whenexecuted by a processor, manage parameters of a semiconductor component,a cell, a library of components, or a library of cells in electronicdesign automation (EDA) software to generate a manufacturable design foran integrated circuit. In implementation, the various componentsdescribed herein might be implemented as discrete components or thefunctions and features described can be shared in part or in total amongone or more components. Aspects of the present invention are describedherein with reference to flowchart illustrations and/or block diagramsof methods, apparatus (systems), and computer program products accordingto embodiments of the invention. It will be understood that each blockof the flowchart illustrations and/or block diagrams, and combinationsof blocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

It will be understood from the foregoing description that modificationsand changes may be made in various embodiments of the present inventionwithout departing from its true spirit. The descriptions in thisspecification are for purposes of illustration only and are not to beconstrued in a limiting sense. The scope of the present disclosure islimited only by the language of the following claims.

What is claimed is:
 1. A method for facilitating human intervention inan autonomous device, the method comprising: selecting from a firstplurality of human interventionists, by a service controller, a firstset of human interventionists to respond to a request associated with anautonomous device; transmitting, by the service controller, the requestto a first set of interventionist devices, each interventionist deviceof the first set of interventionist devices associated with a particularhuman interventionist in the first set of human interventionists; andreceiving from the first set of interventionist devices, by the servicecontroller, a first set of interventionist responses to the request. 2.The method of claim 1 further comprising: based on the first set ofinterventionist responses, generating, by the service controller, aservice response to the request; and transmitting, by the servicecontroller, the service response to the autonomous device.
 3. The methodof claim 1 further comprising: determining, by the service controller,whether the service controller is sufficiently confident in the firstset of interventionist responses; in response to determining that theservice controller is sufficiently confident in the first set ofinterventionist responses: based on the first set of interventionistresponses, generating, by the service controller, a service response tothe request; and transmitting, by the service controller, the serviceresponse to the autonomous device; and in response to determining thatthe service controller is not sufficiently confident in the first set ofinterventionist responses: selecting from a second plurality of humaninterventionists, by the service controller, a second set of humaninterventionists to respond to the request; transmitting, by the servicecontroller, the request to a second set of interventionist devices, eachinterventionist device of the second set of interventionist devicesassociated with a particular human interventionist in the second set ofhuman interventionists; and receiving, by the service controller, asecond set of interventionist responses from the second set ofinterventionist devices.
 4. The method of claim 3 further comprising:determining, by the service controller, whether the service controlleris sufficiently confident in an aggregate of the first set ofinterventionist responses and the second set of interventionistresponses; in response to determining that the service controller issufficiently confident in the aggregate of the first set ofinterventionist responses and the second set of interventionistresponses: based on the aggregate of the first set of interventionistresponses and the second set of interventionist responses, generating,by the service controller, a service response to the request; andtransmitting, by the service controller, the service response to theautonomous device; and in response to determining that the servicecontroller is not sufficiently confident in the aggregate of the firstset of interventionist responses and the second set of interventionistresponses: selecting from a third plurality of human interventionists,by the service controller, a third set of human interventionists torespond to the request; transmitting, by the service controller, therequest to a third set of interventionist devices, each interventionistdevice of the third set of interventionist devices associated with aparticular human interventionist in the third set of humaninterventionists; and receiving, by the service controller, a third setof interventionist responses from the third set of interventionistdevices.
 5. The method of claim 1 further comprises: determining, by theservice controller, whether the service controller can perform therequest autonomously; and wherein selecting from the first plurality ofhuman interventionists, by the service controller, the first set ofhuman interventionists to respond to the request is in response todetermining that the service controller cannot perform the requestautonomously; wherein transmitting, by the service controller, therequest to the first set of interventionist devices is in response todetermining that the service controller cannot perform the requestautonomously; and wherein receiving from the first set ofinterventionist devices, by the service controller, the first set ofinterventionist responses to the request is in response to determiningthat the service controller cannot perform the request autonomously. 6.The method of claim 5 further comprising: in response to determiningthat the service controller can perform the request autonomously,generating, by the service controller, an autonomous response.
 7. Themethod of claim 6 further comprising: based on the autonomous response,generating, by the service controller, a service response to therequest; and transmitting, by the service controller, the serviceresponse to the autonomous device.
 8. The method of claim 1 furthercomprising: determining, by the service controller, whether the servicecontroller can perform the request autonomously; in response todetermining that the service controller can perform the requestautonomously, generating, by the service controller, an autonomousresponse. determining, by the service controller, whether the servicecontroller is sufficiently confident in the autonomous response; inresponse to determining that the service controller is sufficientlyconfident in the autonomous response: based on the autonomous response,generating, by the service controller, a service response to therequest; and transmitting, by the service controller, the serviceresponse to the autonomous device; and wherein selecting from the firstplurality of human interventionists, by the service controller, thefirst set of human interventionists to respond to the request is inresponse to determining that that the service controller is notsufficiently confident in the autonomous response; wherein transmitting,by the service controller, the request to the first set ofinterventionist devices is in response to determining that that theservice controller is not sufficiently confident in the autonomousresponse; and wherein receiving from the first set of interventionistdevices, by the service controller, the first set of interventionistresponses to the request is in response to determining that that theservice controller is not sufficiently confident in the autonomousresponse.
 9. The method of claim 8 further comprising: determining, bythe service controller, whether the service controller is sufficientlyconfident in an aggregate of the autonomous response and the first setof interventionist responses; in response determining that the servicecontroller is sufficiently confident in the aggregate of the autonomousresponse and the first set of interventionist responses: based on one ofthe first set of interventionist response and the aggregate of theautonomous response and the first set of interventionist responses,generating, by the service controller, a service response to therequest; and transmitting, by the service controller, the serviceresponse to the autonomous device; and in response to determining thatthe service controller is not sufficiently confident in the aggregate ofthe autonomous response and the first set of interventionist responses:selecting from a second plurality of human interventionists, by theservice controller, a second set of human interventionists to respond tothe request; transmitting, by the service controller, the request to asecond set of interventionist devices, each interventionist device ofthe second set of interventionist devices associated with a particularhuman interventionist in the second set of human interventionists; andreceiving, by the service controller, a second set of interventionistresponses from the second set of interventionist devices.
 10. The methodof claim 1, wherein selecting from the first plurality of humaninterventionists, by the service controller, the first set of humaninterventionists to respond to the request includes: determining, forthe request, an expense range and selection criteria; for one or morehuman interventionists of the first plurality of human interventionists,determining an interventionist expense for each of the one or more humaninterventionists, the interventionist expense indicating the cost ofusing the human interventionist to respond to the request; andselecting, from the first plurality of human interventionists, the firstset of human interventionists based on the expense range, the selectioncriteria, and the interventionist expense of each of the one or morehuman interventionists.
 11. The method of claim 1, wherein selectingfrom the first plurality of human interventionists, by the servicecontroller, the first set of human interventionists to respond to therequest includes: determining, for the request, a difficulty rating andselection criteria; for one or more human interventionists of the firstplurality of human interventionists, determining a qualification ratingof each of the one or more human interventionists; and selecting, fromthe first plurality of human interventionists, the first set of humaninterventionists based on the difficulty rating, the selection criteria,and the qualification rating of each of the one or more humaninterventionists.
 12. The method of claim 1 wherein selecting from thefirst plurality of human interventionists, by the service controller,the first set of human interventionists to respond to the requestincludes: for one or more human interventionists of the first pluralityof human interventionists, determining a responsiveness of each of theone or more human interventionists; determining, for the request, anurgency rating and selection criteria; and selecting, from the firstplurality of human interventionists, the first set of humaninterventionists based on the urgency rating, the selection criteria,and the responsiveness of each of the one or more humaninterventionists.
 13. The method of claim 1 wherein selecting from thefirst plurality of human interventionists, by the service controller,the first set of human interventionists to respond to the requestincludes: for one or more human interventionists of the first pluralityof human interventionists, determining, by the service controller, aperformance metric of each of the one or more human interventionists;determining, for the request, a performance requirement and selectioncriteria; and selecting, from the first plurality of humaninterventionists, the first set of human interventionists based on theperformance requirement, the selection criteria, and the performancemetric of each of the one or more human interventionists.
 14. The methodof claim 1 wherein selecting from the first plurality of humaninterventionists, by the service controller, the first set of humaninterventionists to respond to the request includes: determining, by theservice controller, a confidence requirement associated with therequest; determining a number of human interventionists needed tosatisfy the confidence requirement; and including at least thedetermined number of human interventionists within the first set ofhuman interventionists.
 15. The method of claim 1 wherein selecting fromthe first plurality of human interventionists, by the servicecontroller, the first set of human interventionists to respond to therequest includes: determining a type of task to be performed, by a humaninterventionist, for responding to the request; determining a type ofinterventionist device for performing the task; and selecting for thefirst set of human interventionists, a particular human interventionisthaving a particular interventionist device matching the determined typeof interventionist device.
 16. The method of claim 1 wherein selectingfrom the first plurality of human interventionists, by the servicecontroller, the first set of human interventionists to respond to therequest includes: determining a type of task to be performed by a humaninterventionist, for responding the request; and selecting for the firstset of human interventionists, a particular human interventionist havinga profile indicating the capability of performing the type of task to beperformed for responding to the request.
 17. The method of claim 1wherein each human interventionist in the first set of humaninterventionists has an associated expense for performing a particulartype of request; the method further comprising: attributing to anaccount associated with the autonomous device, by the servicecontroller, the particular expense associated with each humaninterventionist in the first set of human interventionists.
 18. Themethod of claim 1 further comprising: for one or more humaninterventionists in the first plurality of human interventionists:transmitting, by the service controller, a test request to a particularinterventionist device associated with a particular humaninterventionist; receiving from the particular interventionist device,by the service controller, a test response to the transmitted testrequest; and based on the test response, generating, by the servicecontroller, a performance metric of the human interventionist; and basedon the performance metric of each of the one or more humaninterventionists of the first plurality of human interventionists,determining, by the service controller, a rating for each of the one ormore human interventionists; and wherein selecting from the firstplurality of human interventionists, by the service controller, thefirst set of human interventionists to respond to the request includesselecting, based on the determined ratings, the first set of humaninterventionists.
 19. The method of claim 18, wherein based on the testresponse, generating, by the service controller, the performance metricof the human interventionist includes: transmitting the test response toa human interventionist evaluator; and receiving from the humaninterventionist evaluator, an evaluation of the test response.
 20. Themethod of claim 1 wherein selecting from the first plurality of humaninterventionists, by the service controller, the first set of humaninterventionists to respond to the request includes: determining, by theservice controller, whether a particular human interventionist isavailable; in response to determining that the particular humaninterventionist is not available, excluding, by the service controller,the particular human interventionist from the first set of humaninterventionists to respond to the request; and in response todetermining that the particular human interventionist is available,adding, by the service controller, the particular human interventionistto the first set of human interventionists to respond to the request.21. A computer readable medium for facilitating human intervention in anautonomous device, the computer readable medium including a computerprogram instructions that when executed by a computer processor causethe computer processor to perform: for each human interventionist of aplurality of human interventionists: transmitting, by a servicecontroller, a test request to a particular interventionist deviceassociated with a particular human interventionist; receiving from theparticular interventionist device, by the service controller, a testresponse to the transmitted test request; and based on the testresponse, generating, by the service controller, a performance metric ofthe human interventionist; and based on the performance metrics of thehuman interventionists of the plurality of human interventionists,determining, by the service controller, a rating for each humaninterventionist of the plurality of human interventionists.
 22. Thecomputer readable medium of claim 21 further comprising computer programinstructions that when executed by the computer processor cause thecomputer processor to perform: receiving, by the service controller, arequest from an autonomous device; selecting from the plurality of humaninterventionists, by the service controller, a first set of humaninterventionists to respond to the request; transmitting, by the servicecontroller, the request to a first set of interventionist devices, eachinterventionist device of the first set of interventionist devicesassociated with a particular human interventionist in the first set ofhuman interventionists; and receiving from the first set ofinterventionist devices, by the service controller, a first set ofinterventionist responses to the request.
 23. A method of facilitatinghuman intervention in an autonomous device, the method comprising:placing, by an autonomous device, an object in a stack of one or moreobjects; in response to placing the object in the stack, detecting, bythe autonomous device, instability in the stack; determining, by theautonomous device, that correcting the instability requires a level ofcapability beyond a threshold associated with the autonomous device; andin response to determining that correcting the instability requires thelevel of capability beyond the threshold associated with the autonomousdevice, generating, by the autonomous device, a request for humanintervention.
 24. A method of facilitating human intervention in anautonomous device, the method comprising: grasping, by an autonomousdevice, an object with one or more components of the autonomous device;based on sensor input, determining, by the autonomous device, a level ofconfidence that the object is securely grasped with the one or morecomponents of the autonomous device; determining, by the autonomousdevice, that the level does not satisfy a confidence requirement; and inresponse to determining that the level does not satisfy the confidencerequirement, generating, by the autonomous device, a request for humanintervention.
 25. A method of facilitating human intervention in anautonomous device, the method comprising: receiving, by a servicecontroller, a request from an autonomous device; generating, by theservice controller, a service response to the request, the serviceresponse generated based on least one of: a first set of interventionistresponses received from a first set of interventionist devicesassociated with a first set of human interventionists; and a first setof autonomous responses generated by an autonomous module; and training,by the service controller, the autonomous module using at least one ofthe first set of interventionist responses and the first set ofautonomous responses.
 26. The method of claim 25 further comprisingtransmitting, by the service controller, the service response to theautonomous device.